Sep 11 13:51:00.019635 ip-10-0-136-162 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 11 13:51:00.019707 ip-10-0-136-162 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 11 13:51:00.019717 ip-10-0-136-162 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 11 13:51:00.020070 ip-10-0-136-162 systemd[1]: Failed to start Kubernetes Kubelet. Sep 11 13:51:10.268435 ip-10-0-136-162 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Sep 11 13:51:10.268457 ip-10-0-136-162 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 66906a272d944ae9a089f000b2c4b286 -- Sep 11 13:53:01.643959 ip-10-0-136-162 systemd[1]: Starting Kubernetes Kubelet... Sep 11 13:53:02.092575 ip-10-0-136-162 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:53:02.092575 ip-10-0-136-162 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 11 13:53:02.092575 ip-10-0-136-162 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:53:02.092575 ip-10-0-136-162 kubenswrapper[2579]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 11 13:53:02.092575 ip-10-0-136-162 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 11 13:53:02.093776 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.093693 2579 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 11 13:53:02.096467 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096452 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:53:02.096467 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096467 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096471 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096474 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096477 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096479 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096482 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096484 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096487 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096490 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096493 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096495 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096498 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096500 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096503 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096505 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096508 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096510 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096520 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096522 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096525 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:53:02.096535 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096527 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096530 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096532 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096535 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096538 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096541 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096543 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096546 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096549 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096552 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096554 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096557 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096559 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096562 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096565 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096567 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096570 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096572 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096574 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:53:02.097067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096577 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096580 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096582 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096585 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096588 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096590 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096592 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096594 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096597 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096601 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096605 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096608 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096611 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096613 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096616 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096620 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096624 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096627 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096630 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:53:02.097563 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096633 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096636 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096638 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096641 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096644 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096646 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096662 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096666 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096668 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096671 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096674 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096676 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096679 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096681 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096683 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096686 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096688 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096691 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096693 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096696 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:53:02.098121 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096698 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096702 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096704 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096707 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096712 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096716 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096719 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096723 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096725 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096728 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096731 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096734 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096736 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096739 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096742 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096744 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096747 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096749 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096751 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:53:02.098672 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.096754 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097403 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097411 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097415 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097418 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097421 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097424 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097428 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097432 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097434 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097437 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097440 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097443 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097446 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097448 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097451 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097453 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097456 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097459 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:53:02.099174 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097462 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097464 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097467 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097470 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097472 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097474 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097477 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097480 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097482 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097484 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097487 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097489 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097491 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097494 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097496 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097499 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097501 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097503 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097506 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097508 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:53:02.099700 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097511 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097514 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097516 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097519 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097521 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097524 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097526 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097529 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097531 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097534 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097541 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097543 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097547 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097550 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097552 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097555 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097558 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097560 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097562 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097565 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:53:02.100413 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097568 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097570 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097573 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097575 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097578 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097580 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097583 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097585 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097587 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097590 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097594 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097597 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097600 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097602 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097605 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097608 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097611 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097613 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097616 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:53:02.100966 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097618 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097621 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097623 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097626 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097628 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097631 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097634 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097636 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097639 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097641 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097643 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097646 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097663 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097666 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097669 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097671 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097673 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097676 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097679 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097681 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:53:02.101557 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097684 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.097687 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098039 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098044 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098047 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098063 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098066 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098070 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098074 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098077 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098080 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098083 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098086 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098088 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098091 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098094 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098096 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098099 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098102 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098105 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:53:02.102100 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098108 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098110 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098112 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098115 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098117 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098119 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098122 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098124 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098127 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098130 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098132 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098134 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098137 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098139 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098141 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098144 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098146 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098149 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098151 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098154 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098156 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:53:02.102618 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098159 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098161 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098164 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098166 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098169 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098172 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098174 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098177 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098180 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098182 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098185 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098188 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098191 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098193 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098196 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098198 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098201 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098205 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098207 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:53:02.103183 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098210 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098213 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098216 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098218 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098221 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098223 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098226 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098228 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098230 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098233 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098235 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098238 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098240 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098243 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098245 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098247 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098251 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098254 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098256 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098258 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:53:02.103732 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098261 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098263 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098265 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098268 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098271 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098273 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098276 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098278 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098280 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098283 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098285 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098288 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098290 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098292 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098295 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098297 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098300 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098303 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098307 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:53:02.104262 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098309 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.098313 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098386 2579 flags.go:64] FLAG: --address="0.0.0.0" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098420 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098425 2579 flags.go:64] FLAG: --anonymous-auth="true" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098430 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098434 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098437 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098441 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098445 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098448 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098451 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098454 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098458 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098460 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098463 2579 flags.go:64] FLAG: --cgroup-root="" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098466 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098469 2579 flags.go:64] FLAG: --client-ca-file="" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098472 2579 flags.go:64] FLAG: --cloud-provider="external" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098476 2579 flags.go:64] FLAG: --cluster-dns="[]" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098479 2579 flags.go:64] FLAG: --cluster-domain="" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098482 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098486 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 11 13:53:02.104785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098489 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098492 2579 flags.go:64] FLAG: --container-log-max-files="5" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098496 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098499 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098501 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098505 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098507 2579 flags.go:64] FLAG: --contention-profiling="false" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098510 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098513 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098516 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098519 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098523 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098526 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098531 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098534 2579 flags.go:64] FLAG: --enable-load-reader="false" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098537 2579 flags.go:64] FLAG: --enable-server="true" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098539 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098543 2579 flags.go:64] FLAG: --event-burst="100" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098546 2579 flags.go:64] FLAG: --event-qps="50" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098548 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098551 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098554 2579 flags.go:64] FLAG: --eviction-hard="" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098558 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098561 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098564 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 11 13:53:02.105368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098566 2579 flags.go:64] FLAG: --eviction-soft="" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098569 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098572 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098575 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098578 2579 flags.go:64] FLAG: --experimental-mounter-path="" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098582 2579 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098585 2579 flags.go:64] FLAG: --fail-swap-on="true" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098588 2579 flags.go:64] FLAG: --feature-gates="" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098592 2579 flags.go:64] FLAG: --file-check-frequency="20s" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098594 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098597 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098600 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098603 2579 flags.go:64] FLAG: --healthz-port="10248" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098606 2579 flags.go:64] FLAG: --help="false" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098609 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098611 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098614 2579 flags.go:64] FLAG: --http-check-frequency="20s" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098617 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098620 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098623 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098627 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098630 2579 flags.go:64] FLAG: --image-service-endpoint="" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098633 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 11 13:53:02.106024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098635 2579 flags.go:64] FLAG: --kube-api-burst="100" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098638 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098641 2579 flags.go:64] FLAG: --kube-api-qps="50" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098644 2579 flags.go:64] FLAG: --kube-reserved="" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098646 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098664 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098667 2579 flags.go:64] FLAG: --kubelet-cgroups="" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098670 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098673 2579 flags.go:64] FLAG: --lock-file="" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098676 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098679 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098682 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098687 2579 flags.go:64] FLAG: --log-json-split-stream="false" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098691 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098695 2579 flags.go:64] FLAG: --log-text-split-stream="false" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098698 2579 flags.go:64] FLAG: --logging-format="text" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098701 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098704 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098707 2579 flags.go:64] FLAG: --manifest-url="" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098710 2579 flags.go:64] FLAG: --manifest-url-header="" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098714 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098717 2579 flags.go:64] FLAG: --max-open-files="1000000" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098721 2579 flags.go:64] FLAG: --max-pods="110" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098723 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098726 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 11 13:53:02.106668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098729 2579 flags.go:64] FLAG: --memory-manager-policy="None" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098732 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098735 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098737 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098741 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098748 2579 flags.go:64] FLAG: --node-status-max-images="50" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098751 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098754 2579 flags.go:64] FLAG: --oom-score-adj="-999" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098757 2579 flags.go:64] FLAG: --pod-cidr="" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098760 2579 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098765 2579 flags.go:64] FLAG: --pod-manifest-path="" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098768 2579 flags.go:64] FLAG: --pod-max-pids="-1" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098771 2579 flags.go:64] FLAG: --pods-per-core="0" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098774 2579 flags.go:64] FLAG: --port="10250" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098777 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098780 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d2ad7542047f3994" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098783 2579 flags.go:64] FLAG: --qos-reserved="" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098786 2579 flags.go:64] FLAG: --read-only-port="10255" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098789 2579 flags.go:64] FLAG: --register-node="true" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098791 2579 flags.go:64] FLAG: --register-with-taints="" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098795 2579 flags.go:64] FLAG: --registry-burst="10" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098799 2579 flags.go:64] FLAG: --registry-qps="5" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098802 2579 flags.go:64] FLAG: --reserved-cpus="" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098805 2579 flags.go:64] FLAG: --reserved-memory="" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098809 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 11 13:53:02.107384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098812 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098814 2579 flags.go:64] FLAG: --rotate-certificates="false" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098817 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098820 2579 flags.go:64] FLAG: --runonce="false" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098823 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098825 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098828 2579 flags.go:64] FLAG: --seccomp-default="false" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098831 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098834 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098837 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098840 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098843 2579 flags.go:64] FLAG: --storage-driver-password="root" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098847 2579 flags.go:64] FLAG: --storage-driver-secure="false" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098850 2579 flags.go:64] FLAG: --storage-driver-table="stats" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098852 2579 flags.go:64] FLAG: --storage-driver-user="root" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098855 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098858 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098861 2579 flags.go:64] FLAG: --system-cgroups="" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098864 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098869 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098871 2579 flags.go:64] FLAG: --tls-cert-file="" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098874 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098878 2579 flags.go:64] FLAG: --tls-min-version="" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098880 2579 flags.go:64] FLAG: --tls-private-key-file="" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098883 2579 flags.go:64] FLAG: --topology-manager-policy="none" Sep 11 13:53:02.108387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098886 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098888 2579 flags.go:64] FLAG: --topology-manager-scope="container" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098891 2579 flags.go:64] FLAG: --v="2" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098896 2579 flags.go:64] FLAG: --version="false" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098900 2579 flags.go:64] FLAG: --vmodule="" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098904 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.098907 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099043 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099049 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099052 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099055 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099058 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099061 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099063 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099066 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099068 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099071 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099073 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099076 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099080 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099082 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:53:02.109282 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099084 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099087 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099090 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099093 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099095 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099097 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099100 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099102 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099105 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099107 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099111 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099114 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099117 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099119 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099123 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099126 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099131 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099134 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099136 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:53:02.109859 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099138 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099141 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099143 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099146 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099148 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099151 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099153 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099156 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099158 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099160 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099163 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099167 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099169 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099172 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099174 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099177 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099179 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099182 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099185 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099187 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:53:02.110369 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099190 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099193 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099195 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099198 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099200 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099203 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099205 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099208 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099211 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099213 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099216 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099219 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099221 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099224 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099226 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099229 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099231 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099234 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099236 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099239 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:53:02.110933 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099241 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099244 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099246 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099251 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099254 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099256 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099259 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099261 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099264 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099266 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099269 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099271 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099274 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099276 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099278 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099281 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099283 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099286 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099288 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099292 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:53:02.111490 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099294 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099296 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099300 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099304 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099306 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.099309 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.099906 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.109933 2579 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.109948 2579 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.111402 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.111410 2579 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111467 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111472 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111475 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:53:02.112026 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111478 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111481 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111484 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111487 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111489 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111492 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111495 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111498 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111501 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111505 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111509 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111512 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111516 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111519 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111522 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111525 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111528 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111530 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111533 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:53:02.112411 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111536 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111539 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111541 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111544 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111546 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111549 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111552 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111554 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111557 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111559 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111562 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111564 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111567 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111569 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111571 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111574 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111576 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111579 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111581 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111584 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:53:02.112918 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111587 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111589 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111592 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111594 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111596 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111599 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111602 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111604 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111607 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111609 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111611 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111614 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111617 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111620 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111623 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111625 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111627 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111630 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111632 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111635 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:53:02.113440 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111637 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111640 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111642 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111645 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111647 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111661 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111664 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111666 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111669 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111671 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111674 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111676 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111679 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111681 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111685 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111688 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111691 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111694 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111698 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:53:02.113977 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111701 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111704 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111707 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111710 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111713 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111715 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111718 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111721 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111723 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111726 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111728 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111731 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111733 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111736 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111738 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111741 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111743 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:53:02.114465 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111746 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.111751 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111862 2579 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111867 2579 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111871 2579 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111875 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111878 2579 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111881 2579 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111884 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111887 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111889 2579 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111891 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111894 2579 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111897 2579 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 11 13:53:02.115021 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111900 2579 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111902 2579 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111904 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111907 2579 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111909 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111912 2579 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111914 2579 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111917 2579 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111919 2579 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111922 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111924 2579 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111927 2579 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111930 2579 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111932 2579 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111935 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111937 2579 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111940 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111942 2579 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111945 2579 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111948 2579 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 11 13:53:02.115387 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111950 2579 feature_gate.go:328] unrecognized feature gate: Example Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111953 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111955 2579 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111958 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111961 2579 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111964 2579 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111967 2579 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111970 2579 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111972 2579 feature_gate.go:328] unrecognized feature gate: Example2 Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111975 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111978 2579 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111980 2579 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111983 2579 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111987 2579 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111989 2579 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111992 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111994 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111996 2579 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.111999 2579 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 11 13:53:02.115927 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112001 2579 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112004 2579 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112006 2579 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112011 2579 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112013 2579 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112016 2579 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112019 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112021 2579 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112024 2579 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112026 2579 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112029 2579 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112031 2579 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112034 2579 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112037 2579 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112039 2579 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112042 2579 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112044 2579 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112046 2579 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112057 2579 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112060 2579 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 11 13:53:02.116429 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112062 2579 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112065 2579 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112068 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112070 2579 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112073 2579 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112075 2579 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112078 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112081 2579 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112084 2579 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112087 2579 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112090 2579 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112094 2579 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112097 2579 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112099 2579 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112101 2579 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112105 2579 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112108 2579 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112111 2579 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112113 2579 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112116 2579 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 11 13:53:02.116954 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112118 2579 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112121 2579 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112123 2579 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112126 2579 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112128 2579 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112131 2579 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112133 2579 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:02.112136 2579 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.112141 2579 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.112242 2579 server.go:962] "Client rotation is on, will bootstrap in background" Sep 11 13:53:02.117512 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.116709 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 11 13:53:02.117861 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.117849 2579 server.go:1019] "Starting client certificate rotation" Sep 11 13:53:02.117974 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.117960 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 13:53:02.118007 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.117996 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 11 13:53:02.121916 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.121902 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 13:53:02.123088 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.123071 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 11 13:53:02.134054 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.134035 2579 log.go:25] "Validated CRI v1 runtime API" Sep 11 13:53:02.138669 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.138640 2579 log.go:25] "Validated CRI v1 image API" Sep 11 13:53:02.139544 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.139529 2579 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 11 13:53:02.141601 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.141583 2579 fs.go:135] Filesystem UUIDs: map[29421a23-e143-4e38-a32d-bde2a348eb6e:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 b5292f0e-d508-4f3e-9a96-b91aa898ac36:/dev/nvme0n1p3] Sep 11 13:53:02.141688 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.141600 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 11 13:53:02.148772 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.148635 2579 manager.go:216] Machine: {Timestamp:2026-09-11 13:53:02.146829924 +0000 UTC m=+0.369009747 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090995 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21a4a4bc2b87cde757650452b8960a SystemUUID:ec21a4a4-bc2b-87cd-e757-650452b8960a BootID:66906a27-2d94-4ae9-a089-f000b2c4b286 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:6c:59:94:83:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:6c:59:94:83:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:db:1f:57:fd:e1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 11 13:53:02.148772 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.148760 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 11 13:53:02.148929 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.148906 2579 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 11 13:53:02.154002 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.153975 2579 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 11 13:53:02.154154 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.154004 2579 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-162.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 11 13:53:02.155176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.155164 2579 topology_manager.go:138] "Creating topology manager with none policy" Sep 11 13:53:02.155239 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.155183 2579 container_manager_linux.go:309] "Creating device plugin manager" Sep 11 13:53:02.155239 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.155193 2579 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 13:53:02.155239 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.155216 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 11 13:53:02.156064 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.156044 2579 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 11 13:53:02.159789 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.159776 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 13:53:02.159915 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.159904 2579 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 11 13:53:02.159971 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.159945 2579 kubelet.go:486] "Attempting to sync node with API server" Sep 11 13:53:02.159971 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.159960 2579 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 11 13:53:02.160070 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.159978 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 11 13:53:02.160070 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.159992 2579 kubelet.go:398] "Adding apiserver pod source" Sep 11 13:53:02.160070 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.160004 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 11 13:53:02.160386 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.160374 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 13:53:02.160448 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.160398 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 11 13:53:02.162954 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.162135 2579 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 11 13:53:02.162954 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.163046 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 13:53:02.162954 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.163715 2579 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 11 13:53:02.162954 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.163742 2579 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165258 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165272 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165278 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165283 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165291 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165296 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165309 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165314 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165320 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165325 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165333 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.165338 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.166676 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 11 13:53:02.168702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.166684 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 11 13:53:02.170272 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.170260 2579 server.go:1268] "Started kubelet" Sep 11 13:53:02.170475 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.170438 2579 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 11 13:53:02.170894 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.170859 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 11 13:53:02.170991 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.170903 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 11 13:53:02.170958 ip-10-0-136-162 systemd[1]: Started Kubernetes Kubelet. Sep 11 13:53:02.172835 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.172818 2579 server.go:310] "Adding debug handlers to kubelet server" Sep 11 13:53:02.173208 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.173188 2579 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 11 13:53:02.176881 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.176866 2579 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 11 13:53:02.177380 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.177363 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-162.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 11 13:53:02.178086 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.178062 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 11 13:53:02.178167 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.178060 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 11 13:53:02.178909 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.178896 2579 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 11 13:53:02.178976 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.178911 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 11 13:53:02.178976 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.178965 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.179647 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179630 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 11 13:53:02.179743 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179630 2579 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 11 13:53:02.179743 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179684 2579 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179773 2579 reconstruct.go:97] "Volume reconstruction finished" Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179781 2579 reconciler.go:29] "Reconciler: start to sync state" Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179845 2579 factory.go:153] Registering CRI-O factory Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179860 2579 factory.go:223] Registration of the crio container factory successfully Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179922 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179931 2579 factory.go:55] Registering systemd factory Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179937 2579 factory.go:223] Registration of the systemd container factory successfully Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179953 2579 factory.go:103] Registering Raw factory Sep 11 13:53:02.179993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.179964 2579 manager.go:1196] Started watching for new ooms in manager Sep 11 13:53:02.180388 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.180242 2579 manager.go:349] Starting recovery of all containers Sep 11 13:53:02.180388 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.177361 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-162.ec2.internal.18d4489aea8d92bb default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-162.ec2.internal,UID:ip-10-0-136-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-162.ec2.internal,},FirstTimestamp:2026-09-11 13:53:02.170239675 +0000 UTC m=+0.392419496,LastTimestamp:2026-09-11 13:53:02.170239675 +0000 UTC m=+0.392419496,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-162.ec2.internal,}" Sep 11 13:53:02.187568 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.187536 2579 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 11 13:53:02.187714 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.187671 2579 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-162.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 11 13:53:02.192787 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.192771 2579 manager.go:354] Recovery completed Sep 11 13:53:02.197006 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.196992 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.197809 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.197794 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ghjh7" Sep 11 13:53:02.198608 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.198593 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.198684 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.198619 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.198684 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.198630 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.199175 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199163 2579 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 11 13:53:02.199175 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199175 2579 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 11 13:53:02.199277 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199188 2579 state_mem.go:36] "Initialized new in-memory state store" Sep 11 13:53:02.199277 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199191 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.199963 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199945 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.200047 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199979 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.200047 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.199993 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.200869 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.200801 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-162.ec2.internal.18d4489aec3e6cf2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-162.ec2.internal,UID:ip-10-0-136-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-162.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-162.ec2.internal,},FirstTimestamp:2026-09-11 13:53:02.19860709 +0000 UTC m=+0.420786911,LastTimestamp:2026-09-11 13:53:02.19860709 +0000 UTC m=+0.420786911,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-162.ec2.internal,}" Sep 11 13:53:02.201377 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.201363 2579 policy_none.go:49] "None policy: Start" Sep 11 13:53:02.201436 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.201384 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 11 13:53:02.201436 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.201397 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 11 13:53:02.203407 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.203395 2579 policy_none.go:47] "Start" Sep 11 13:53:02.208960 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.208867 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ghjh7" Sep 11 13:53:02.210742 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.210687 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-162.ec2.internal.18d4489aec3ead94 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-162.ec2.internal,UID:ip-10-0-136-162.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-162.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-162.ec2.internal,},FirstTimestamp:2026-09-11 13:53:02.198623636 +0000 UTC m=+0.420803457,LastTimestamp:2026-09-11 13:53:02.198623636 +0000 UTC m=+0.420803457,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-162.ec2.internal,}" Sep 11 13:53:02.245199 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245186 2579 manager.go:337] "Starting Device Plugin manager" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.245217 2579 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245226 2579 server.go:87] "Starting device plugin registration server" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245446 2579 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245456 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245558 2579 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245736 2579 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.245745 2579 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.246099 2579 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 11 13:53:02.264618 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.246140 2579 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.282270 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.282245 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 11 13:53:02.283544 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.283532 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 11 13:53:02.283613 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.283576 2579 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 11 13:53:02.283613 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.283595 2579 kubelet.go:2448] "Starting kubelet main sync loop" Sep 11 13:53:02.283754 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.283666 2579 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 11 13:53:02.287357 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.287343 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:53:02.345889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.345836 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.346558 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.346537 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.346632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.346571 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.346632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.346586 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.346632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.346606 2579 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.361248 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.361233 2579 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.361290 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.361251 2579 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-162.ec2.internal\": node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.384128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.384112 2579 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal"] Sep 11 13:53:02.384173 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.384161 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.385492 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.385480 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.385554 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.385504 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.385554 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.385519 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.387885 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.387874 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.388010 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.387996 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.388046 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388026 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.388526 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388510 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.388595 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388525 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.388595 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388539 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.388595 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388546 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.388595 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388556 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.388797 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.388557 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.390803 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.390785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.390890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.390811 2579 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 11 13:53:02.391488 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.391472 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientMemory" Sep 11 13:53:02.391569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.391504 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasNoDiskPressure" Sep 11 13:53:02.391569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.391519 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeHasSufficientPID" Sep 11 13:53:02.415166 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.415142 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-162.ec2.internal\" not found" node="ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.419527 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.419509 2579 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-162.ec2.internal\" not found" node="ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.481662 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.481629 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd8fb514ac9220f7e4129e6bad38217c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal\" (UID: \"dd8fb514ac9220f7e4129e6bad38217c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.481732 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.481668 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8fb514ac9220f7e4129e6bad38217c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal\" (UID: \"dd8fb514ac9220f7e4129e6bad38217c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.481732 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.481683 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a570e7183fc751dd6686e350e76db83-config\") pod \"kube-apiserver-proxy-ip-10-0-136-162.ec2.internal\" (UID: \"1a570e7183fc751dd6686e350e76db83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.490012 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.489994 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.582072 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.582053 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd8fb514ac9220f7e4129e6bad38217c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal\" (UID: \"dd8fb514ac9220f7e4129e6bad38217c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.582130 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.582086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8fb514ac9220f7e4129e6bad38217c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal\" (UID: \"dd8fb514ac9220f7e4129e6bad38217c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.582130 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.582108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a570e7183fc751dd6686e350e76db83-config\") pod \"kube-apiserver-proxy-ip-10-0-136-162.ec2.internal\" (UID: \"1a570e7183fc751dd6686e350e76db83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.582130 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.582118 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/dd8fb514ac9220f7e4129e6bad38217c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal\" (UID: \"dd8fb514ac9220f7e4129e6bad38217c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.582219 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.582147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1a570e7183fc751dd6686e350e76db83-config\") pod \"kube-apiserver-proxy-ip-10-0-136-162.ec2.internal\" (UID: \"1a570e7183fc751dd6686e350e76db83\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.582219 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.582159 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/dd8fb514ac9220f7e4129e6bad38217c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal\" (UID: \"dd8fb514ac9220f7e4129e6bad38217c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.590698 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.590678 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.691207 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.691161 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.717593 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.717571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.722622 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:02.722552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" Sep 11 13:53:02.791695 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.791673 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.892147 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.892128 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:02.992694 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:02.992647 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:03.072192 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.072173 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:53:03.092955 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:03.092938 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:03.117076 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.117055 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 11 13:53:03.117197 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.117180 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:53:03.117248 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.117179 2579 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 11 13:53:03.193713 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:03.193689 2579 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-136-162.ec2.internal\" not found" Sep 11 13:53:03.213993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.213957 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:48:03 +0000 UTC" deadline="2028-04-28 11:28:13.097616869 +0000 UTC" Sep 11 13:53:03.213993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.213990 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14277h35m9.88363178s" Sep 11 13:53:03.265520 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.265501 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 13:53:03.272908 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.272890 2579 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:53:03.280078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.280059 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" Sep 11 13:53:03.286868 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.286831 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" event={"ID":"1a570e7183fc751dd6686e350e76db83","Type":"ContainerStarted","Data":"f3652ffd806deff1003b969766e6034c1d1cc66855ab3c889be96a6cc90bb173"} Sep 11 13:53:03.300710 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.300693 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 13:53:03.301755 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.301744 2579 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" Sep 11 13:53:03.311834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.311818 2579 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 11 13:53:03.355345 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:03.355323 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8fb514ac9220f7e4129e6bad38217c.slice/crio-42d605df8b7ee000eb5e19236582529470ed55b56286e6f90c6fe85688e7e2dc WatchSource:0}: Error finding container 42d605df8b7ee000eb5e19236582529470ed55b56286e6f90c6fe85688e7e2dc: Status 404 returned error can't find the container with id 42d605df8b7ee000eb5e19236582529470ed55b56286e6f90c6fe85688e7e2dc Sep 11 13:53:03.516248 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:03.516196 2579 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:53:05.449537 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.449509 2579 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:53:05.513898 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.513877 2579 apiserver.go:52] "Watching apiserver" Sep 11 13:53:05.532745 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.532717 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 11 13:53:05.533823 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.532993 2579 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 11 13:53:05.533823 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.533483 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m","openshift-multus/multus-additional-cni-plugins-5cgnq","openshift-multus/network-metrics-daemon-zbh64","openshift-network-diagnostics/network-check-target-wl6rx","openshift-network-operator/iptables-alerter-fr2xg","openshift-ovn-kubernetes/ovnkube-node-w4pl7","kube-system/global-pull-secret-syncer-nxrhp","kube-system/konnectivity-agent-2dk4m","kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9l8zc","openshift-dns/node-resolver-kkhl9","openshift-image-registry/node-ca-jttfb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal","openshift-multus/multus-542kv"] Sep 11 13:53:05.535961 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.535942 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.538488 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.538464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.539096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.539075 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.539239 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.539180 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.539566 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.539549 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zsp4t\"" Sep 11 13:53:05.540656 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.540635 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.542651 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.542629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.544488 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.544467 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 11 13:53:05.544582 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.544509 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 11 13:53:05.544582 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.544514 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.544698 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.544635 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.544949 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.544909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:05.545025 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.544973 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:05.545080 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545021 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-x9jjt\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545378 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-47k4j\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545558 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545578 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545608 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545772 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545930 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-smcc8\"" Sep 11 13:53:05.546150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.545967 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.546543 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.546459 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 11 13:53:05.552857 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.552838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.554549 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.554367 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.555301 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.555282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.555406 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.555343 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:05.558457 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.558429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.561012 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.560992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.563326 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.563306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.565896 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.565875 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 11 13:53:05.566123 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.566079 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.566281 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.566265 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 11 13:53:05.566905 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.566889 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 11 13:53:05.567180 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.567084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-hj8l8\"" Sep 11 13:53:05.567452 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.567433 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-w9dsn\"" Sep 11 13:53:05.567618 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.567604 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.567939 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.567912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-542kv" Sep 11 13:53:05.568148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.568127 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:05.568234 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.568135 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6622v\"" Sep 11 13:53:05.568234 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.568212 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:05.568340 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.568258 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 11 13:53:05.568463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.568446 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 11 13:53:05.568549 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.568471 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 11 13:53:05.568729 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.568713 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.572052 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.572034 2579 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 11 13:53:05.572736 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.572716 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 11 13:53:05.573212 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.573194 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4nczq\"" Sep 11 13:53:05.573494 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.573476 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.573743 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.573725 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-dqwrz\"" Sep 11 13:53:05.573851 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.573833 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 11 13:53:05.574892 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.574872 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 11 13:53:05.578568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.578550 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 11 13:53:05.601002 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.600980 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mk9cs" Sep 11 13:53:05.633031 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.633016 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 11 13:53:05.637805 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.637790 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mk9cs" Sep 11 13:53:05.640684 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640664 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysconfig\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.640834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640695 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-sys\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.640834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d275ee-fd61-4460-8bb1-013e6f6d08d2-tmp-dir\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.640834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640741 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-netns\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.640834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovnkube-config\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.640834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640817 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovnkube-script-lib\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-modprobe-d\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640865 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-kubernetes\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640888 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640912 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-etc-kubernetes\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsl5z\" (UniqueName: \"kubernetes.io/projected/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-kube-api-access-bsl5z\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640959 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pwqb\" (UniqueName: \"kubernetes.io/projected/764fc16c-8069-4014-94cc-199ddca114a6-kube-api-access-8pwqb\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.640982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysctl-d\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-systemd\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6rl9\" (UniqueName: \"kubernetes.io/projected/c22deef1-6981-4597-9bf2-b7d39471a64f-kube-api-access-m6rl9\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641081 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-cni-bin\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641206 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-cni-binary-copy\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641235 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-kubelet\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641260 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-var-lib-kubelet\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-var-lib-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641345 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c22deef1-6981-4597-9bf2-b7d39471a64f-tmp\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.641447 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641453 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-device-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641478 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbnf\" (UniqueName: \"kubernetes.io/projected/06d275ee-fd61-4460-8bb1-013e6f6d08d2-kube-api-access-9wbnf\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxp9n\" (UniqueName: \"kubernetes.io/projected/9036a4b6-21fd-47ea-9092-556af9ba0a7b-kube-api-access-kxp9n\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641525 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-systemd-units\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-node-log\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641573 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-cni-netd\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-socket-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641607 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-sys-fs\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-system-cni-dir\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-slash\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b3eaae6-ad54-476c-8449-f8b9ab33a196-dbus\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-cni-multus\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641721 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641761 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96df2f98-53ac-4384-b273-e57016a3ec89-konnectivity-ca\") pod \"konnectivity-agent-2dk4m\" (UID: \"96df2f98-53ac-4384-b273-e57016a3ec89\") " pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.641848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641785 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-lib-modules\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-cnibin\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641825 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9036a4b6-21fd-47ea-9092-556af9ba0a7b-serviceca\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641879 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-ovn\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641899 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysctl-conf\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.641939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a63cac-0cb0-410d-9e83-6260d4df9ed8-iptables-alerter-script\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-conf-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-multus-certs\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-kubelet\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642088 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-cni-bin\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642128 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-run\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642154 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-cni-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642180 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9036a4b6-21fd-47ea-9092-556af9ba0a7b-host\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642219 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhgm8\" (UniqueName: \"kubernetes.io/projected/84e79f72-14a6-4317-963f-6fd7f359bba0-kube-api-access-xhgm8\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96df2f98-53ac-4384-b273-e57016a3ec89-agent-certs\") pod \"konnectivity-agent-2dk4m\" (UID: \"96df2f98-53ac-4384-b273-e57016a3ec89\") " pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:05.642463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642303 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-host\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-tuned\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642356 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovn-node-metrics-cert\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642377 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfwgp\" (UniqueName: \"kubernetes.io/projected/0f50aa33-480c-469f-b7f5-055a1c03cd5a-kube-api-access-tfwgp\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642400 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06d275ee-fd61-4460-8bb1-013e6f6d08d2-hosts-file\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642425 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-system-cni-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-daemon-config\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642488 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-etc-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqs9q\" (UniqueName: \"kubernetes.io/projected/8d081982-c02f-4f76-a910-bd0406b66e19-kube-api-access-qqs9q\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642533 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a63cac-0cb0-410d-9e83-6260d4df9ed8-host-slash\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qz5l\" (UniqueName: \"kubernetes.io/projected/34a63cac-0cb0-410d-9e83-6260d4df9ed8-kube-api-access-7qz5l\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642606 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" event={"ID":"dd8fb514ac9220f7e4129e6bad38217c","Type":"ContainerStarted","Data":"42d605df8b7ee000eb5e19236582529470ed55b56286e6f90c6fe85688e7e2dc"} Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642622 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-k8s-cni-cncf-io\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642649 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642672 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-run-netns\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642705 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-systemd\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.643087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642727 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-log-socket\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642742 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-registration-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-socket-dir-parent\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642787 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-hostroot\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642830 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-cnibin\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b3eaae6-ad54-476c-8449-f8b9ab33a196-kubelet-config\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.643580 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.642915 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-env-overrides\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.743480 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743403 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-cni-multus\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.743480 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.743480 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743473 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96df2f98-53ac-4384-b273-e57016a3ec89-konnectivity-ca\") pod \"konnectivity-agent-2dk4m\" (UID: \"96df2f98-53ac-4384-b273-e57016a3ec89\") " pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743524 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-cni-multus\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743532 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-lib-modules\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-cnibin\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9036a4b6-21fd-47ea-9092-556af9ba0a7b-serviceca\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:05.743704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-ovn\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743723 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysctl-conf\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743731 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-lib-modules\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a63cac-0cb0-410d-9e83-6260d4df9ed8-iptables-alerter-script\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-conf-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743783 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-multus-certs\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-kubelet\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-cni-bin\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-run\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-cni-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9036a4b6-21fd-47ea-9092-556af9ba0a7b-host\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhgm8\" (UniqueName: \"kubernetes.io/projected/84e79f72-14a6-4317-963f-6fd7f359bba0-kube-api-access-xhgm8\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743882 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96df2f98-53ac-4384-b273-e57016a3ec89-agent-certs\") pod \"konnectivity-agent-2dk4m\" (UID: \"96df2f98-53ac-4384-b273-e57016a3ec89\") " pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:05.743930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-host\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743941 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-tuned\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovn-node-metrics-cert\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743956 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/96df2f98-53ac-4384-b273-e57016a3ec89-konnectivity-ca\") pod \"konnectivity-agent-2dk4m\" (UID: \"96df2f98-53ac-4384-b273-e57016a3ec89\") " pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfwgp\" (UniqueName: \"kubernetes.io/projected/0f50aa33-480c-469f-b7f5-055a1c03cd5a-kube-api-access-tfwgp\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06d275ee-fd61-4460-8bb1-013e6f6d08d2-hosts-file\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-system-cni-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744034 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-daemon-config\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-etc-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744049 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-cni-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqs9q\" (UniqueName: \"kubernetes.io/projected/8d081982-c02f-4f76-a910-bd0406b66e19-kube-api-access-qqs9q\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a63cac-0cb0-410d-9e83-6260d4df9ed8-host-slash\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qz5l\" (UniqueName: \"kubernetes.io/projected/34a63cac-0cb0-410d-9e83-6260d4df9ed8-kube-api-access-7qz5l\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744122 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-k8s-cni-cncf-io\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysctl-conf\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-run-netns\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.744426 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744214 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-systemd\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-log-socket\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-registration-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-socket-dir-parent\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744312 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-hostroot\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744339 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-cnibin\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9036a4b6-21fd-47ea-9092-556af9ba0a7b-serviceca\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-etc-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744482 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b3eaae6-ad54-476c-8449-f8b9ab33a196-kubelet-config\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-env-overrides\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysconfig\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-sys\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744624 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d275ee-fd61-4460-8bb1-013e6f6d08d2-tmp-dir\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-netns\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovnkube-config\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744692 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/34a63cac-0cb0-410d-9e83-6260d4df9ed8-iptables-alerter-script\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.745176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744699 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovnkube-script-lib\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-modprobe-d\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744728 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-kubernetes\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744742 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744746 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-conf-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744756 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-etc-kubernetes\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744796 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsl5z\" (UniqueName: \"kubernetes.io/projected/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-kube-api-access-bsl5z\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744798 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-multus-certs\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744838 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-kubelet\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pwqb\" (UniqueName: \"kubernetes.io/projected/764fc16c-8069-4014-94cc-199ddca114a6-kube-api-access-8pwqb\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysctl-d\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-systemd\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6rl9\" (UniqueName: \"kubernetes.io/projected/c22deef1-6981-4597-9bf2-b7d39471a64f-kube-api-access-m6rl9\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-cni-bin\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744991 2579 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745023 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/34a63cac-0cb0-410d-9e83-6260d4df9ed8-host-slash\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-cni-binary-copy\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.745859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-kubelet\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745120 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9036a4b6-21fd-47ea-9092-556af9ba0a7b-host\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745155 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-var-lib-kubelet\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745202 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5b3eaae6-ad54-476c-8449-f8b9ab33a196-kubelet-config\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745209 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-var-lib-kubelet\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-var-lib-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745302 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-registration-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c22deef1-6981-4597-9bf2-b7d39471a64f-tmp\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.745408 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-cni-binary-copy\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745431 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-device-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.745479 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-systemd\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.744076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-ovn\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.746068 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:06.24601978 +0000 UTC m=+3.115751034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:05.746686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbnf\" (UniqueName: \"kubernetes.io/projected/06d275ee-fd61-4460-8bb1-013e6f6d08d2-kube-api-access-9wbnf\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746569 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-host\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746321 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-cnibin\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746407 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-k8s-cni-cncf-io\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746439 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.746154 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.743959 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-cnibin\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.746673 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:06.246655091 +0000 UTC m=+3.116386351 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746189 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-socket-dir-parent\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.746812 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-cni-binary-copy\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.747464 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.747117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-env-overrides\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.749160 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-run\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.749255 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxp9n\" (UniqueName: \"kubernetes.io/projected/9036a4b6-21fd-47ea-9092-556af9ba0a7b-kube-api-access-kxp9n\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.749255 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749229 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysconfig\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.749255 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-systemd-units\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.749255 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-sysctl-d\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.749472 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749292 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-node-log\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.749472 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-systemd-units\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.749587 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-cni-netd\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.749643 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749516 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-socket-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.749643 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-sys-fs\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.749741 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749651 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/06d275ee-fd61-4460-8bb1-013e6f6d08d2-hosts-file\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.749741 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749661 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-system-cni-dir\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.749840 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749740 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/764fc16c-8069-4014-94cc-199ddca114a6-system-cni-dir\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.749840 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749415 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-cni-netd\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.749840 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749776 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-system-cni-dir\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.749840 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/96df2f98-53ac-4384-b273-e57016a3ec89-agent-certs\") pod \"konnectivity-agent-2dk4m\" (UID: \"96df2f98-53ac-4384-b273-e57016a3ec89\") " pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.750040 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749458 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-node-log\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.750040 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749870 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-socket-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.750040 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749935 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-sys-fs\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.750040 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.749971 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-hostroot\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.750040 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750019 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-kubelet\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.750309 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750075 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.750309 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750141 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-var-lib-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.750309 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750200 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-var-lib-cni-bin\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.750446 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.750499 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750461 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-systemd\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.750546 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750513 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-log-socket\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.750914 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750884 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-run-ovn-kubernetes\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.750994 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750963 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-run-openvswitch\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.751053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.750982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovnkube-script-lib\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.751134 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751058 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-kubernetes\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.751134 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751067 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-modprobe-d\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.751134 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751069 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-multus-daemon-config\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.751285 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751156 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-host-run-netns\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.751285 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751180 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c22deef1-6981-4597-9bf2-b7d39471a64f-sys\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.751285 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751228 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-cni-bin\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.751430 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06d275ee-fd61-4460-8bb1-013e6f6d08d2-tmp-dir\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.751477 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751450 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751548 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-etc-kubernetes\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/84e79f72-14a6-4317-963f-6fd7f359bba0-device-dir\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovnkube-config\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751869 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-slash\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b3eaae6-ad54-476c-8449-f8b9ab33a196-dbus\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.752054 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5b3eaae6-ad54-476c-8449-f8b9ab33a196-dbus\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751801 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-run-netns\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.752134 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0f50aa33-480c-469f-b7f5-055a1c03cd5a-host-slash\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.752181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.751828 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c22deef1-6981-4597-9bf2-b7d39471a64f-tmp\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.752709 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.752314 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/764fc16c-8069-4014-94cc-199ddca114a6-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.753669 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.753646 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c22deef1-6981-4597-9bf2-b7d39471a64f-etc-tuned\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.753770 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.753713 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0f50aa33-480c-469f-b7f5-055a1c03cd5a-ovn-node-metrics-cert\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.757456 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.757437 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:05.757562 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.757461 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:05.757562 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.757474 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:05.757562 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:05.757526 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:53:06.257509747 +0000 UTC m=+3.127241010 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:05.778143 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.778117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qz5l\" (UniqueName: \"kubernetes.io/projected/34a63cac-0cb0-410d-9e83-6260d4df9ed8-kube-api-access-7qz5l\") pod \"iptables-alerter-fr2xg\" (UID: \"34a63cac-0cb0-410d-9e83-6260d4df9ed8\") " pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.778261 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.778237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhgm8\" (UniqueName: \"kubernetes.io/projected/84e79f72-14a6-4317-963f-6fd7f359bba0-kube-api-access-xhgm8\") pod \"aws-ebs-csi-driver-node-l8c7m\" (UID: \"84e79f72-14a6-4317-963f-6fd7f359bba0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.781321 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.781300 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqs9q\" (UniqueName: \"kubernetes.io/projected/8d081982-c02f-4f76-a910-bd0406b66e19-kube-api-access-qqs9q\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:05.792828 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.792754 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxp9n\" (UniqueName: \"kubernetes.io/projected/9036a4b6-21fd-47ea-9092-556af9ba0a7b-kube-api-access-kxp9n\") pod \"node-ca-jttfb\" (UID: \"9036a4b6-21fd-47ea-9092-556af9ba0a7b\") " pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.794699 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.794422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbnf\" (UniqueName: \"kubernetes.io/projected/06d275ee-fd61-4460-8bb1-013e6f6d08d2-kube-api-access-9wbnf\") pod \"node-resolver-kkhl9\" (UID: \"06d275ee-fd61-4460-8bb1-013e6f6d08d2\") " pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.795818 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.795794 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfwgp\" (UniqueName: \"kubernetes.io/projected/0f50aa33-480c-469f-b7f5-055a1c03cd5a-kube-api-access-tfwgp\") pod \"ovnkube-node-w4pl7\" (UID: \"0f50aa33-480c-469f-b7f5-055a1c03cd5a\") " pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.797145 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.797117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsl5z\" (UniqueName: \"kubernetes.io/projected/326c14d7-7a75-403a-a7bf-5e0b7d07a4d2-kube-api-access-bsl5z\") pod \"multus-542kv\" (UID: \"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2\") " pod="openshift-multus/multus-542kv" Sep 11 13:53:05.797490 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.797468 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6rl9\" (UniqueName: \"kubernetes.io/projected/c22deef1-6981-4597-9bf2-b7d39471a64f-kube-api-access-m6rl9\") pod \"tuned-9l8zc\" (UID: \"c22deef1-6981-4597-9bf2-b7d39471a64f\") " pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.798523 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.798505 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pwqb\" (UniqueName: \"kubernetes.io/projected/764fc16c-8069-4014-94cc-199ddca114a6-kube-api-access-8pwqb\") pod \"multus-additional-cni-plugins-5cgnq\" (UID: \"764fc16c-8069-4014-94cc-199ddca114a6\") " pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.854057 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.854029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" Sep 11 13:53:05.861589 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.861565 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" Sep 11 13:53:05.869452 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.869431 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jttfb" Sep 11 13:53:05.878080 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.878057 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" Sep 11 13:53:05.885785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.885767 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:05.892562 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.892543 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:05.901156 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.901135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fr2xg" Sep 11 13:53:05.907763 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.907742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-kkhl9" Sep 11 13:53:05.914277 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:05.914254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-542kv" Sep 11 13:53:06.255989 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.255949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:06.256190 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.256029 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:06.256190 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.256134 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:06.256190 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.256165 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:06.256357 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.256212 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:07.256190323 +0000 UTC m=+4.125921567 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:06.256357 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.256233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:07.256223255 +0000 UTC m=+4.125954500 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:06.356939 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.356906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:06.357081 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.357032 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:06.357081 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.357048 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:06.357081 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.357061 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:06.357236 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.357134 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:53:07.35711966 +0000 UTC m=+4.226850914 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:06.448430 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.448406 2579 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 11 13:53:06.636993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.636964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:06.637445 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:06.637151 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:06.641158 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.641130 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:48:05 +0000 UTC" deadline="2028-02-28 06:33:39.949947987 +0000 UTC" Sep 11 13:53:06.641158 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:06.641158 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12832h40m33.308794333s" Sep 11 13:53:06.696191 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.696162 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f50aa33_480c_469f_b7f5_055a1c03cd5a.slice/crio-7d3c49ff8ca0f200f6c7df2cb8cc8d08da3b37af59cc2f1fb82f50cc8b71d0d7 WatchSource:0}: Error finding container 7d3c49ff8ca0f200f6c7df2cb8cc8d08da3b37af59cc2f1fb82f50cc8b71d0d7: Status 404 returned error can't find the container with id 7d3c49ff8ca0f200f6c7df2cb8cc8d08da3b37af59cc2f1fb82f50cc8b71d0d7 Sep 11 13:53:06.701597 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.701574 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a63cac_0cb0_410d_9e83_6260d4df9ed8.slice/crio-96d6af0dcdcff3bb8a17730c4e7379938bf0c71dee63a07a3718e7a227086fbd WatchSource:0}: Error finding container 96d6af0dcdcff3bb8a17730c4e7379938bf0c71dee63a07a3718e7a227086fbd: Status 404 returned error can't find the container with id 96d6af0dcdcff3bb8a17730c4e7379938bf0c71dee63a07a3718e7a227086fbd Sep 11 13:53:06.703001 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.702818 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22deef1_6981_4597_9bf2_b7d39471a64f.slice/crio-111c5d9a078b122759b4ada74d40b3d5acd5149bf2895ae4d52ad569367845b9 WatchSource:0}: Error finding container 111c5d9a078b122759b4ada74d40b3d5acd5149bf2895ae4d52ad569367845b9: Status 404 returned error can't find the container with id 111c5d9a078b122759b4ada74d40b3d5acd5149bf2895ae4d52ad569367845b9 Sep 11 13:53:06.703193 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.703089 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9036a4b6_21fd_47ea_9092_556af9ba0a7b.slice/crio-a1b54e20adc93713c06d90879e4903e845d1087a522c8d8b0b6c33c27ca59b83 WatchSource:0}: Error finding container a1b54e20adc93713c06d90879e4903e845d1087a522c8d8b0b6c33c27ca59b83: Status 404 returned error can't find the container with id a1b54e20adc93713c06d90879e4903e845d1087a522c8d8b0b6c33c27ca59b83 Sep 11 13:53:06.706276 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.706252 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96df2f98_53ac_4384_b273_e57016a3ec89.slice/crio-64d69a2a48a57eac2b14074b82c72c008c3fe78f6184d6c67f4fef13cd7fad9a WatchSource:0}: Error finding container 64d69a2a48a57eac2b14074b82c72c008c3fe78f6184d6c67f4fef13cd7fad9a: Status 404 returned error can't find the container with id 64d69a2a48a57eac2b14074b82c72c008c3fe78f6184d6c67f4fef13cd7fad9a Sep 11 13:53:06.707655 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.707410 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84e79f72_14a6_4317_963f_6fd7f359bba0.slice/crio-b03239786f759323689b601ed1e89d8a12b03051c283686a60abfe18e5c60702 WatchSource:0}: Error finding container b03239786f759323689b601ed1e89d8a12b03051c283686a60abfe18e5c60702: Status 404 returned error can't find the container with id b03239786f759323689b601ed1e89d8a12b03051c283686a60abfe18e5c60702 Sep 11 13:53:06.709212 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.709189 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d275ee_fd61_4460_8bb1_013e6f6d08d2.slice/crio-f90a331b12f18b5308d09c5e10648b6b8b6d3ba2385f78ed2d0ce486e9ba4ceb WatchSource:0}: Error finding container f90a331b12f18b5308d09c5e10648b6b8b6d3ba2385f78ed2d0ce486e9ba4ceb: Status 404 returned error can't find the container with id f90a331b12f18b5308d09c5e10648b6b8b6d3ba2385f78ed2d0ce486e9ba4ceb Sep 11 13:53:06.710309 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:06.710277 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod764fc16c_8069_4014_94cc_199ddca114a6.slice/crio-2992db1e0a6cac16880f87183214e8b8b375358e5d5d8d5c17ea7ce88cd49d8c WatchSource:0}: Error finding container 2992db1e0a6cac16880f87183214e8b8b375358e5d5d8d5c17ea7ce88cd49d8c: Status 404 returned error can't find the container with id 2992db1e0a6cac16880f87183214e8b8b375358e5d5d8d5c17ea7ce88cd49d8c Sep 11 13:53:07.265485 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.265241 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:07.265619 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.265407 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:07.265619 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.265557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:07.265731 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.265626 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:09.265603103 +0000 UTC m=+6.135334346 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:07.265731 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.265721 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:07.265846 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.265779 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:09.265760951 +0000 UTC m=+6.135492207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:07.365898 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.365861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:07.366087 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.366066 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:07.366192 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.366093 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:07.366192 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.366125 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:07.366192 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.366181 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:53:09.366163312 +0000 UTC m=+6.235894566 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:07.639406 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.639233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:07.639406 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.639233 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:07.639406 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.639362 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:07.640292 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:07.639436 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:07.641542 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.641478 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:48:05 +0000 UTC" deadline="2028-03-29 19:37:47.596549984 +0000 UTC" Sep 11 13:53:07.641542 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.641523 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13565h44m39.955030621s" Sep 11 13:53:07.667625 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.667593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" event={"ID":"84e79f72-14a6-4317-963f-6fd7f359bba0","Type":"ContainerStarted","Data":"b03239786f759323689b601ed1e89d8a12b03051c283686a60abfe18e5c60702"} Sep 11 13:53:07.674915 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.674888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2dk4m" event={"ID":"96df2f98-53ac-4384-b273-e57016a3ec89","Type":"ContainerStarted","Data":"64d69a2a48a57eac2b14074b82c72c008c3fe78f6184d6c67f4fef13cd7fad9a"} Sep 11 13:53:07.687894 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.687844 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-542kv" event={"ID":"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2","Type":"ContainerStarted","Data":"afac04fd9ac09d6f5852c4efdf78b713c376fc5edc80f990d3ec1abcb738ffc8"} Sep 11 13:53:07.701361 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.701336 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" event={"ID":"c22deef1-6981-4597-9bf2-b7d39471a64f","Type":"ContainerStarted","Data":"111c5d9a078b122759b4ada74d40b3d5acd5149bf2895ae4d52ad569367845b9"} Sep 11 13:53:07.714187 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.714161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"7d3c49ff8ca0f200f6c7df2cb8cc8d08da3b37af59cc2f1fb82f50cc8b71d0d7"} Sep 11 13:53:07.721143 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.720374 2579 generic.go:356] "Generic (PLEG): container finished" podID="dd8fb514ac9220f7e4129e6bad38217c" containerID="e8295bd92b0e705e9e9b3b64e2a4543aa8e791dacbf9b563b0803e3f6d79d39d" exitCode=0 Sep 11 13:53:07.721143 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.720446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" event={"ID":"dd8fb514ac9220f7e4129e6bad38217c","Type":"ContainerDied","Data":"e8295bd92b0e705e9e9b3b64e2a4543aa8e791dacbf9b563b0803e3f6d79d39d"} Sep 11 13:53:07.729340 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.729283 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerStarted","Data":"2992db1e0a6cac16880f87183214e8b8b375358e5d5d8d5c17ea7ce88cd49d8c"} Sep 11 13:53:07.760209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.760181 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jttfb" event={"ID":"9036a4b6-21fd-47ea-9092-556af9ba0a7b","Type":"ContainerStarted","Data":"a1b54e20adc93713c06d90879e4903e845d1087a522c8d8b0b6c33c27ca59b83"} Sep 11 13:53:07.767303 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.767247 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fr2xg" event={"ID":"34a63cac-0cb0-410d-9e83-6260d4df9ed8","Type":"ContainerStarted","Data":"96d6af0dcdcff3bb8a17730c4e7379938bf0c71dee63a07a3718e7a227086fbd"} Sep 11 13:53:07.793824 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.793788 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" event={"ID":"1a570e7183fc751dd6686e350e76db83","Type":"ContainerStarted","Data":"c3c54da41af428f6123402f64774be648d6ccbc23ab9a6a4018a95c28b00859d"} Sep 11 13:53:07.811131 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.811064 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-162.ec2.internal" podStartSLOduration=3.811048276 podStartE2EDuration="3.811048276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:07.810288759 +0000 UTC m=+4.680020022" watchObservedRunningTime="2026-09-11 13:53:07.811048276 +0000 UTC m=+4.680779540" Sep 11 13:53:07.817522 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:07.817473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kkhl9" event={"ID":"06d275ee-fd61-4460-8bb1-013e6f6d08d2","Type":"ContainerStarted","Data":"f90a331b12f18b5308d09c5e10648b6b8b6d3ba2385f78ed2d0ce486e9ba4ceb"} Sep 11 13:53:08.636874 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:08.636843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:08.637044 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:08.636982 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:08.830012 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:08.829972 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" event={"ID":"dd8fb514ac9220f7e4129e6bad38217c","Type":"ContainerStarted","Data":"fcad9a4bcb875305af996b538ffa860d31ff6932ca5ca1af3a82fee82042bbc6"} Sep 11 13:53:09.281987 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:09.281946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:09.282181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:09.282030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:09.282257 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.282233 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:09.282316 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.282293 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:13.282276212 +0000 UTC m=+10.152007467 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:09.282708 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.282689 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:09.282800 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.282747 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:13.282730619 +0000 UTC m=+10.152461875 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:09.382857 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:09.382820 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:09.383474 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.383030 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:09.383474 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.383056 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:09.383474 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.383068 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:09.383474 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.383141 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:53:13.383124387 +0000 UTC m=+10.252855651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:09.639508 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:09.638881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:09.639508 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.639003 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:09.639508 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:09.639062 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:09.639508 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:09.639169 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:10.637184 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:10.637154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:10.637548 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:10.637277 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:11.638490 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:11.638462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:11.638873 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:11.638555 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:11.638964 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:11.638890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:11.639013 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:11.638957 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:12.636863 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:12.636377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:12.636863 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:12.636516 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:13.323810 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:13.323778 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:13.324290 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:13.323834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:13.324290 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.323939 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:13.324290 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.323996 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:21.323979248 +0000 UTC m=+18.193710501 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:13.324459 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.324335 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:13.324459 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.324376 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:21.324361207 +0000 UTC m=+18.194092461 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:13.424334 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:13.424284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:13.424526 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.424423 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:13.424526 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.424444 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:13.424526 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.424453 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:13.424526 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.424519 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:53:21.42450221 +0000 UTC m=+18.294233464 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:13.637782 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:13.637348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:13.637782 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.637447 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:13.638057 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:13.638032 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:13.638194 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:13.638150 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:14.636411 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:14.636375 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:14.636896 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:14.636541 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:15.637358 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:15.637315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:15.637796 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:15.637452 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:15.637796 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:15.637526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:15.637796 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:15.637618 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:16.636808 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:16.636777 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:16.636995 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:16.636901 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:17.636609 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:17.636577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:17.637039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:17.636577 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:17.637039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:17.636706 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:17.637039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:17.636767 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:18.636331 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:18.636303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:18.636509 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:18.636420 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:19.638706 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:19.638677 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:19.639048 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:19.638770 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:19.639084 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:19.639061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:19.639188 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:19.639161 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:20.636713 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:20.636678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:20.636933 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:20.636801 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:21.381363 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:21.381324 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:21.381786 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:21.381398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:21.381786 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.381473 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:21.381786 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.381523 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:21.381786 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.381536 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:37.381519564 +0000 UTC m=+34.251250824 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:21.381786 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.381574 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:37.381557565 +0000 UTC m=+34.251288819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:21.481985 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:21.481954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:21.482173 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.482152 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:21.482241 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.482176 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:21.482241 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.482188 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:21.482341 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.482251 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:53:37.482233865 +0000 UTC m=+34.351965120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:21.637209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:21.637139 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:21.637209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:21.637162 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:21.637402 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.637232 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:21.637402 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:21.637318 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:22.636461 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:22.636411 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:22.636927 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:22.636570 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:23.637798 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:23.637764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:23.638217 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:23.637856 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:23.638217 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:23.637926 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:23.638217 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:23.638034 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:24.636762 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:24.636735 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:24.636900 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:24.636831 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:25.637060 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.636772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:25.637661 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.636782 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:25.637661 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:25.637342 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:25.637661 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:25.637453 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:25.862802 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.862748 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-542kv" event={"ID":"326c14d7-7a75-403a-a7bf-5e0b7d07a4d2","Type":"ContainerStarted","Data":"d657c0c8c947872fb871993dfb96814bcdb2a69386411f78d7734f71624fc019"} Sep 11 13:53:25.864485 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.864455 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" event={"ID":"c22deef1-6981-4597-9bf2-b7d39471a64f","Type":"ContainerStarted","Data":"12876701ea316be823cd6977aad9391bd86385c5d342734f0f53be0829e4e3e9"} Sep 11 13:53:25.868300 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.868274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"da50666ddb822225e37fc03654f9d20a9d9d5d03f8069988664497dfac2cc6cc"} Sep 11 13:53:25.868391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.868335 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"2bb842c31e2b3174bb13698a48ed47eaa0b7a98a04e31836af7f03c1d6678a17"} Sep 11 13:53:25.868391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.868373 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"27315914c3cb0f896eaa7cf9774a664024d3d47b00ce5e48ad2dbfcda83df72e"} Sep 11 13:53:25.868517 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.868402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"de80941702e08e707bf7e289ecec7d2f143c69ef62a0ba2e4c989becdae1eef2"} Sep 11 13:53:25.868517 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.868430 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"a30bc655c5fd9d4c834bd2d6a404ab18746451f576e3de2cd3ad818613550e95"} Sep 11 13:53:25.868517 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.868470 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"98f9c053ded804598ddb7897faa45701999e8af6451b3a0eee7bf9e4f501c512"} Sep 11 13:53:25.870255 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.870234 2579 generic.go:356] "Generic (PLEG): container finished" podID="764fc16c-8069-4014-94cc-199ddca114a6" containerID="7dd47120d4a228133623f2514f2fbe059c52996d04470e7115b1d75d93d26391" exitCode=0 Sep 11 13:53:25.870349 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.870313 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerDied","Data":"7dd47120d4a228133623f2514f2fbe059c52996d04470e7115b1d75d93d26391"} Sep 11 13:53:25.871733 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.871713 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jttfb" event={"ID":"9036a4b6-21fd-47ea-9092-556af9ba0a7b","Type":"ContainerStarted","Data":"b5b60e157597e7ed67d893baedda2ce239bdf7635c4404694b95f5abd8cc8111"} Sep 11 13:53:25.873873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.873854 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-kkhl9" event={"ID":"06d275ee-fd61-4460-8bb1-013e6f6d08d2","Type":"ContainerStarted","Data":"c20f5718022dbe0a84d47d51ce8f279f114abcac8d436fa9caacd0054d2538af"} Sep 11 13:53:25.875534 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.875511 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" event={"ID":"84e79f72-14a6-4317-963f-6fd7f359bba0","Type":"ContainerStarted","Data":"111865f0e37e621dae4cf038d7df4877ab8eaf7b3b5b12081f5f0b03231f4464"} Sep 11 13:53:25.876956 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.876933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-2dk4m" event={"ID":"96df2f98-53ac-4384-b273-e57016a3ec89","Type":"ContainerStarted","Data":"4b530c3f2c5c289a21596da9b0b676b9d79a7ef571d924c2496fb979c72b22b6"} Sep 11 13:53:25.902927 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.902884 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-162.ec2.internal" podStartSLOduration=21.902867411 podStartE2EDuration="21.902867411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:08.854483876 +0000 UTC m=+5.724215139" watchObservedRunningTime="2026-09-11 13:53:25.902867411 +0000 UTC m=+22.772598674" Sep 11 13:53:25.903038 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.902972 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-542kv" podStartSLOduration=3.669994485 podStartE2EDuration="21.902965668s" totalImagesPullingTime="18.232971183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:25.902708386 +0000 UTC m=+22.772439649" watchObservedRunningTime="2026-09-11 13:53:25.902965668 +0000 UTC m=+22.772696933" Sep 11 13:53:25.931034 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.930987 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-kkhl9" podStartSLOduration=3.765096974 podStartE2EDuration="21.930972171s" totalImagesPullingTime="18.165875197s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:25.930595275 +0000 UTC m=+22.800326537" watchObservedRunningTime="2026-09-11 13:53:25.930972171 +0000 UTC m=+22.800703446" Sep 11 13:53:25.980810 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:25.980761 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-2dk4m" podStartSLOduration=11.383523176 podStartE2EDuration="21.980743673s" totalImagesPullingTime="10.597220497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:25.980285217 +0000 UTC m=+22.850016480" watchObservedRunningTime="2026-09-11 13:53:25.980743673 +0000 UTC m=+22.850474937" Sep 11 13:53:26.004419 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.004372 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9l8zc" podStartSLOduration=4.804041972 podStartE2EDuration="23.004359646s" totalImagesPullingTime="18.200317674s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:26.004146626 +0000 UTC m=+22.873877888" watchObservedRunningTime="2026-09-11 13:53:26.004359646 +0000 UTC m=+22.874090908" Sep 11 13:53:26.025717 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.025684 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jttfb" podStartSLOduration=4.855105765 podStartE2EDuration="23.025672879s" totalImagesPullingTime="18.170567114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:26.02553199 +0000 UTC m=+22.895263253" watchObservedRunningTime="2026-09-11 13:53:26.025672879 +0000 UTC m=+22.895404141" Sep 11 13:53:26.072814 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.072787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:26.401413 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.401391 2579 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 11 13:53:26.609007 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.608925 2579 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:53:26.40140766Z","UUID":"48484380-a802-421e-963d-d12da186390a","Handler":null,"Name":"","Endpoint":""} Sep 11 13:53:26.610507 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.610485 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 11 13:53:26.610600 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.610515 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 11 13:53:26.636973 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.636950 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:26.637065 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:26.637050 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:26.880364 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.880296 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fr2xg" event={"ID":"34a63cac-0cb0-410d-9e83-6260d4df9ed8","Type":"ContainerStarted","Data":"b09738024fa5a5ed150cb0c6e416f651c4cb97028fb1d9b2da5e2d1c009db3f9"} Sep 11 13:53:26.882703 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.882633 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" event={"ID":"84e79f72-14a6-4317-963f-6fd7f359bba0","Type":"ContainerStarted","Data":"30f8f3c791683cbe9a9561c123647ff6ca8c59ef6db6965ea4e4d9f9b6ba8ca3"} Sep 11 13:53:26.900243 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:26.898326 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fr2xg" podStartSLOduration=4.725071434 podStartE2EDuration="22.898311303s" totalImagesPullingTime="18.173239869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:26.897266938 +0000 UTC m=+23.766998201" watchObservedRunningTime="2026-09-11 13:53:26.898311303 +0000 UTC m=+23.768042574" Sep 11 13:53:27.394039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:27.393977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:27.637178 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:27.637154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:27.637301 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:27.637270 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:27.637533 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:27.637154 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:27.637639 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:27.637597 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:27.890297 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:27.890266 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"b6425f1be59d34b8e8b432cd39448eb308335db0333b1579342b04acca9f38bf"} Sep 11 13:53:28.636766 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.636550 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:28.636919 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:28.636872 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:28.736890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.736860 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:28.737522 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.737501 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:28.895633 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.895538 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" event={"ID":"84e79f72-14a6-4317-963f-6fd7f359bba0","Type":"ContainerStarted","Data":"ef8461ca968ea766d61d14640ada20dea1a9c5941404375cae42ca78218fabe5"} Sep 11 13:53:28.896124 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.895743 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:28.896284 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.896265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-2dk4m" Sep 11 13:53:28.953509 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:28.953468 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8c7m" podStartSLOduration=5.176769275 podStartE2EDuration="25.953454383s" totalImagesPullingTime="20.776685108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:28.930968074 +0000 UTC m=+25.800699336" watchObservedRunningTime="2026-09-11 13:53:28.953454383 +0000 UTC m=+25.823185644" Sep 11 13:53:29.035743 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:29.035704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:29.637230 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:29.637201 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:29.637230 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:29.637234 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:29.637457 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:29.637328 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:29.637511 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:29.637443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:30.636749 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:30.636720 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:30.637154 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:30.636855 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:30.906139 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:30.906075 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" event={"ID":"0f50aa33-480c-469f-b7f5-055a1c03cd5a","Type":"ContainerStarted","Data":"7ef5e40ec732e64eb455656e83017e2e32fa3f02fb192df56050a0e2a04ce2f9"} Sep 11 13:53:30.943686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:30.943645 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" podStartSLOduration=8.699169523 podStartE2EDuration="26.943630453s" totalImagesPullingTime="18.24446093s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:30.943494459 +0000 UTC m=+27.813225722" watchObservedRunningTime="2026-09-11 13:53:30.943630453 +0000 UTC m=+27.813361715" Sep 11 13:53:31.636769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.636608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:31.637245 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.636663 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:31.637245 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:31.636867 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:31.637245 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:31.636896 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:31.909823 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.909754 2579 generic.go:356] "Generic (PLEG): container finished" podID="764fc16c-8069-4014-94cc-199ddca114a6" containerID="3a247e2ec233d6d6e71cf83c7549ffb37bdefe1bd2b8c19a847da2e66bad8e36" exitCode=0 Sep 11 13:53:31.909963 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.909839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerDied","Data":"3a247e2ec233d6d6e71cf83c7549ffb37bdefe1bd2b8c19a847da2e66bad8e36"} Sep 11 13:53:31.910293 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.910269 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:31.910421 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.910337 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:31.910421 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.910386 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:31.926078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.926055 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:31.926200 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:31.926173 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:53:32.628451 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.628405 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:32.637004 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.636981 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:32.637541 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:32.637080 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:32.725319 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.725286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:32.756480 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.756447 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nxrhp"] Sep 11 13:53:32.756637 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.756575 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:32.756699 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:32.756679 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:32.761495 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.761421 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wl6rx"] Sep 11 13:53:32.761605 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.761529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:32.761667 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:32.761622 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:32.762517 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.762493 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zbh64"] Sep 11 13:53:32.826212 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.826053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:32.914764 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.914738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerStarted","Data":"1e1c631c3700a50e32c146dfd2d5896a7ec501166122a6f061eb19bcb8c24ba7"} Sep 11 13:53:32.914764 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.914756 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:32.914902 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:32.914874 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:32.952769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:32.952730 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:33.918879 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:33.918848 2579 generic.go:356] "Generic (PLEG): container finished" podID="764fc16c-8069-4014-94cc-199ddca114a6" containerID="1e1c631c3700a50e32c146dfd2d5896a7ec501166122a6f061eb19bcb8c24ba7" exitCode=0 Sep 11 13:53:33.919292 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:33.918936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerDied","Data":"1e1c631c3700a50e32c146dfd2d5896a7ec501166122a6f061eb19bcb8c24ba7"} Sep 11 13:53:34.637119 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:34.636971 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:34.637260 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:34.637018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:34.637260 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:34.637210 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:34.637395 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:34.637302 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:34.637395 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:34.637018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:34.637483 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:34.637403 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:34.924058 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:34.924025 2579 generic.go:356] "Generic (PLEG): container finished" podID="764fc16c-8069-4014-94cc-199ddca114a6" containerID="548412b2a59341b9eadfa4a34409b57a825bec6f245bc6bf696cf92e756ddb44" exitCode=0 Sep 11 13:53:34.924414 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:34.924082 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerDied","Data":"548412b2a59341b9eadfa4a34409b57a825bec6f245bc6bf696cf92e756ddb44"} Sep 11 13:53:35.333655 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:35.333613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:35.445336 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:35.445286 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:36.636807 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:36.636775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:36.637251 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:36.636888 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:36.637251 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:36.636893 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wl6rx" podUID="7515a898-fe35-45cc-a38f-e1e14009910c" Sep 11 13:53:36.637251 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:36.636894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:36.637251 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:36.637007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-zbh64" podUID="8d081982-c02f-4f76-a910-bd0406b66e19" Sep 11 13:53:36.637251 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:36.637136 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-nxrhp" podUID="5b3eaae6-ad54-476c-8449-f8b9ab33a196" Sep 11 13:53:37.119808 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.119772 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:37.181629 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.181521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:37.397215 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.397136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:37.397215 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.397192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:37.397518 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.397296 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:37.397518 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.397301 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:37.397518 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.397356 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret podName:5b3eaae6-ad54-476c-8449-f8b9ab33a196 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.397340252 +0000 UTC m=+66.267071495 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret") pod "global-pull-secret-syncer-nxrhp" (UID: "5b3eaae6-ad54-476c-8449-f8b9ab33a196") : object "kube-system"/"original-pull-secret" not registered Sep 11 13:53:37.397518 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.397381 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.397364529 +0000 UTC m=+66.267095768 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 11 13:53:37.497919 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.497885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:37.498085 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.498011 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 11 13:53:37.498085 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.498027 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 11 13:53:37.498085 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.498035 2579 projected.go:196] Error preparing data for projected volume kube-api-access-fxp5x for pod openshift-network-diagnostics/network-check-target-wl6rx: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:37.498085 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:37.498088 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x podName:7515a898-fe35-45cc-a38f-e1e14009910c nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.498072796 +0000 UTC m=+66.367804037 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-fxp5x" (UniqueName: "kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x") pod "network-check-target-wl6rx" (UID: "7515a898-fe35-45cc-a38f-e1e14009910c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 11 13:53:37.927821 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.927791 2579 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-136-162.ec2.internal" event="NodeReady" Sep 11 13:53:37.928207 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.927916 2579 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 11 13:53:37.982797 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:37.982717 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-g8b9r"] Sep 11 13:53:38.009459 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.009432 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jgljh"] Sep 11 13:53:38.009621 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.009567 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.012605 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.012578 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 11 13:53:38.013209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.013190 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nllbd\"" Sep 11 13:53:38.013209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.013220 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 11 13:53:38.022249 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.022230 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g8b9r"] Sep 11 13:53:38.022345 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.022263 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgljh"] Sep 11 13:53:38.022345 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.022332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.024826 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.024804 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 11 13:53:38.024919 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.024855 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 11 13:53:38.024919 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.024873 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t9pwf\"" Sep 11 13:53:38.025023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.024919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 11 13:53:38.102280 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.102250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/429e5452-0512-4330-8cf1-4581154d1055-config-volume\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.102456 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.102324 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqtrx\" (UniqueName: \"kubernetes.io/projected/429e5452-0512-4330-8cf1-4581154d1055-kube-api-access-tqtrx\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.102456 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.102344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.102456 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.102397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429e5452-0512-4330-8cf1-4581154d1055-tmp-dir\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.130987 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.130944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:38.203354 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429e5452-0512-4330-8cf1-4581154d1055-tmp-dir\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.203528 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp6n5\" (UniqueName: \"kubernetes.io/projected/596d001e-42a1-409b-95dd-71995ca53f2e-kube-api-access-cp6n5\") pod \"ingress-canary-jgljh\" (UID: \"596d001e-42a1-409b-95dd-71995ca53f2e\") " pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.203528 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/429e5452-0512-4330-8cf1-4581154d1055-config-volume\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.203528 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqtrx\" (UniqueName: \"kubernetes.io/projected/429e5452-0512-4330-8cf1-4581154d1055-kube-api-access-tqtrx\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.203691 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203542 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.203691 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203584 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/596d001e-42a1-409b-95dd-71995ca53f2e-cert\") pod \"ingress-canary-jgljh\" (UID: \"596d001e-42a1-409b-95dd-71995ca53f2e\") " pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.203794 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:38.203705 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:53:38.203794 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:38.203766 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:38.703746803 +0000 UTC m=+35.573478043 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:53:38.203901 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.203806 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/429e5452-0512-4330-8cf1-4581154d1055-tmp-dir\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.204078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.204053 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/429e5452-0512-4330-8cf1-4581154d1055-config-volume\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.215400 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.215380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqtrx\" (UniqueName: \"kubernetes.io/projected/429e5452-0512-4330-8cf1-4581154d1055-kube-api-access-tqtrx\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.230917 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.230873 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:38.304899 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.304859 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp6n5\" (UniqueName: \"kubernetes.io/projected/596d001e-42a1-409b-95dd-71995ca53f2e-kube-api-access-cp6n5\") pod \"ingress-canary-jgljh\" (UID: \"596d001e-42a1-409b-95dd-71995ca53f2e\") " pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.305067 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.304969 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/596d001e-42a1-409b-95dd-71995ca53f2e-cert\") pod \"ingress-canary-jgljh\" (UID: \"596d001e-42a1-409b-95dd-71995ca53f2e\") " pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.308139 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.308087 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/596d001e-42a1-409b-95dd-71995ca53f2e-cert\") pod \"ingress-canary-jgljh\" (UID: \"596d001e-42a1-409b-95dd-71995ca53f2e\") " pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.314811 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.314787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp6n5\" (UniqueName: \"kubernetes.io/projected/596d001e-42a1-409b-95dd-71995ca53f2e-kube-api-access-cp6n5\") pod \"ingress-canary-jgljh\" (UID: \"596d001e-42a1-409b-95dd-71995ca53f2e\") " pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.331781 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.331754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jgljh" Sep 11 13:53:38.637003 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.636912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:53:38.637176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.636912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:53:38.637176 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.636912 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:53:38.640023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.640001 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 13:53:38.640148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.640028 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k94vs\"" Sep 11 13:53:38.640148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.640030 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-spk9p\"" Sep 11 13:53:38.640148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.640035 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 11 13:53:38.640916 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.640891 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 11 13:53:38.641063 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.640893 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 13:53:38.706996 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:38.706957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:38.707212 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:38.707134 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:53:38.707267 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:38.707219 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:39.707197239 +0000 UTC m=+36.576928483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:53:39.639952 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:39.639736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jgljh"] Sep 11 13:53:39.643136 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:53:39.643088 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod596d001e_42a1_409b_95dd_71995ca53f2e.slice/crio-90ef3b6a5264b27ff4148317ad36fec082d284a558e731652737ed3d323c7a70 WatchSource:0}: Error finding container 90ef3b6a5264b27ff4148317ad36fec082d284a558e731652737ed3d323c7a70: Status 404 returned error can't find the container with id 90ef3b6a5264b27ff4148317ad36fec082d284a558e731652737ed3d323c7a70 Sep 11 13:53:39.714766 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:39.714741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:39.714867 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:39.714854 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:53:39.714913 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:39.714905 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:41.714891872 +0000 UTC m=+38.584623111 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:53:39.938879 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:39.938796 2579 generic.go:356] "Generic (PLEG): container finished" podID="764fc16c-8069-4014-94cc-199ddca114a6" containerID="b4744a4125a01b838b0a6a04125d5235d3af0a922c35265113434c2b6194b547" exitCode=0 Sep 11 13:53:39.938879 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:39.938858 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerDied","Data":"b4744a4125a01b838b0a6a04125d5235d3af0a922c35265113434c2b6194b547"} Sep 11 13:53:39.943389 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:39.940607 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgljh" event={"ID":"596d001e-42a1-409b-95dd-71995ca53f2e","Type":"ContainerStarted","Data":"90ef3b6a5264b27ff4148317ad36fec082d284a558e731652737ed3d323c7a70"} Sep 11 13:53:40.947368 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:40.947331 2579 generic.go:356] "Generic (PLEG): container finished" podID="764fc16c-8069-4014-94cc-199ddca114a6" containerID="26f1123355e1d3b181d667d6cf2fad67a6534680c8957130e02d550c4bd1fc13" exitCode=0 Sep 11 13:53:40.948000 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:40.947388 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerDied","Data":"26f1123355e1d3b181d667d6cf2fad67a6534680c8957130e02d550c4bd1fc13"} Sep 11 13:53:41.731247 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:41.731219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:41.731430 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:41.731409 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:53:41.731504 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:41.731487 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:45.731470455 +0000 UTC m=+42.601201701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:53:41.952615 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:41.952585 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" event={"ID":"764fc16c-8069-4014-94cc-199ddca114a6","Type":"ContainerStarted","Data":"ff57f8d3b6f1ab03f6ebaf2d15ec5e03723dd2fdabbba8f4885b295475d4e77e"} Sep 11 13:53:41.981762 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:41.981673 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5cgnq" podStartSLOduration=9.461519167 podStartE2EDuration="38.98165348s" totalImagesPullingTime="29.520134313s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:41.980312886 +0000 UTC m=+38.850044147" watchObservedRunningTime="2026-09-11 13:53:41.98165348 +0000 UTC m=+38.851384742" Sep 11 13:53:42.107962 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:42.107929 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:42.207444 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:42.207413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:42.956287 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:42.956252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jgljh" event={"ID":"596d001e-42a1-409b-95dd-71995ca53f2e","Type":"ContainerStarted","Data":"60270a38c709097114fd7b7843b66a78c3d0456d3ea08649389de780d100f161"} Sep 11 13:53:42.973925 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:42.973883 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jgljh" podStartSLOduration=3.460326416 podStartE2EDuration="5.973867819s" totalImagesPullingTime="2.513541403s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:42.97322898 +0000 UTC m=+39.842960244" watchObservedRunningTime="2026-09-11 13:53:42.973867819 +0000 UTC m=+39.843599084" Sep 11 13:53:43.180520 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:43.180488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:43.243938 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:43.243841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:45.759951 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:45.759915 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:45.760364 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:45.760036 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:53:45.760364 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:45.760091 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:53:53.760075945 +0000 UTC m=+50.629807186 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:53:53.807750 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:53.807716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:53:53.808181 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:53.807827 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:53:53.808181 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:53:53.807892 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.807874457 +0000 UTC m=+66.677605709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:53:56.790193 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:56.790158 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:53:56.974345 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:53:56.974319 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:01.298888 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.298856 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mx52h"] Sep 11 13:54:01.327975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.327951 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mx52h"] Sep 11 13:54:01.328129 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.328033 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" Sep 11 13:54:01.330835 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.330811 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-pj4vc\"" Sep 11 13:54:01.354160 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.354138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/b22cf3e5-3c45-414e-8e59-22b25708f356-kube-api-access-qxp2k\") pod \"network-check-source-6f8b768455-mx52h\" (UID: \"b22cf3e5-3c45-414e-8e59-22b25708f356\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" Sep 11 13:54:01.454385 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.454358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/b22cf3e5-3c45-414e-8e59-22b25708f356-kube-api-access-qxp2k\") pod \"network-check-source-6f8b768455-mx52h\" (UID: \"b22cf3e5-3c45-414e-8e59-22b25708f356\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" Sep 11 13:54:01.493628 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.493609 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxp2k\" (UniqueName: \"kubernetes.io/projected/b22cf3e5-3c45-414e-8e59-22b25708f356-kube-api-access-qxp2k\") pod \"network-check-source-6f8b768455-mx52h\" (UID: \"b22cf3e5-3c45-414e-8e59-22b25708f356\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" Sep 11 13:54:01.539810 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.539781 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4"] Sep 11 13:54:01.564321 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.564274 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s"] Sep 11 13:54:01.564446 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.564417 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" Sep 11 13:54:01.567269 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.567251 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-fl4nv\"" Sep 11 13:54:01.567269 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.567260 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Sep 11 13:54:01.567428 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.567256 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:54:01.586357 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.586327 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-f54f4c8f9-bk2d7"] Sep 11 13:54:01.586778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.586434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:01.595221 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.595201 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 11 13:54:01.595760 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.595742 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 11 13:54:01.595810 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.595797 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:54:01.595854 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.595836 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jpj5q\"" Sep 11 13:54:01.608569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.608545 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4"] Sep 11 13:54:01.608569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.608571 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s"] Sep 11 13:54:01.608730 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.608587 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f54f4c8f9-bk2d7"] Sep 11 13:54:01.608730 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.608675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.612017 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.611990 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 11 13:54:01.612119 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.611994 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 11 13:54:01.612119 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.612061 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bjrq5\"" Sep 11 13:54:01.612119 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.611995 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 11 13:54:01.618882 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.618863 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 11 13:54:01.619981 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.619962 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv"] Sep 11 13:54:01.636655 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.636633 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" Sep 11 13:54:01.640513 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.640494 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.644008 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.643987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv"] Sep 11 13:54:01.647561 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.647543 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Sep 11 13:54:01.647646 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.647547 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Sep 11 13:54:01.647901 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.647886 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-bkmsb\"" Sep 11 13:54:01.647956 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.647891 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:54:01.648393 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.648379 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Sep 11 13:54:01.655999 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.655981 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5b6k\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-kube-api-access-c5b6k\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656066 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-installation-pull-secrets\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656133 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656205 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj2w6\" (UniqueName: \"kubernetes.io/projected/10d8c0d5-cd9a-4ff7-a719-ea1ac937a986-kube-api-access-dj2w6\") pod \"volume-data-source-validator-86695b7cc7-ljsn4\" (UID: \"10d8c0d5-cd9a-4ff7-a719-ea1ac937a986\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" Sep 11 13:54:01.656205 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pt6s\" (UniqueName: \"kubernetes.io/projected/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-kube-api-access-8pt6s\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:01.656282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-image-registry-private-configuration\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656250 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:01.656282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-bound-sa-token\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656413 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656285 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c49eeabe-0790-4d42-9541-ade2388aacde-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.656413 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656320 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c49eeabe-0790-4d42-9541-ade2388aacde-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.656413 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-trusted-ca\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656495 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2lbq\" (UniqueName: \"kubernetes.io/projected/c49eeabe-0790-4d42-9541-ade2388aacde-kube-api-access-z2lbq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.656568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-ca-trust-extracted\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.656568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.656559 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-certificates\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.714819 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.714788 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj"] Sep 11 13:54:01.732206 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.732182 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj"] Sep 11 13:54:01.732342 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.732289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.734980 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.734958 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 11 13:54:01.735425 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.735405 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-6njtx\"" Sep 11 13:54:01.735511 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.735408 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 11 13:54:01.735556 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.735505 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:54:01.735556 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.735537 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 11 13:54:01.737394 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.737360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:01.757138 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757224 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-serving-cert\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.757224 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757179 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxfkw\" (UniqueName: \"kubernetes.io/projected/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-kube-api-access-dxfkw\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.757224 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:01.757205 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:54:01.757224 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:01.757218 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f54f4c8f9-bk2d7: secret "image-registry-tls" not found Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj2w6\" (UniqueName: \"kubernetes.io/projected/10d8c0d5-cd9a-4ff7-a719-ea1ac937a986-kube-api-access-dj2w6\") pod \"volume-data-source-validator-86695b7cc7-ljsn4\" (UID: \"10d8c0d5-cd9a-4ff7-a719-ea1ac937a986\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:01.757262 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls podName:ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:02.257244983 +0000 UTC m=+59.126976223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls") pod "image-registry-f54f4c8f9-bk2d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7") : secret "image-registry-tls" not found Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757322 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pt6s\" (UniqueName: \"kubernetes.io/projected/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-kube-api-access-8pt6s\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-image-registry-private-configuration\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757374 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-bound-sa-token\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c49eeabe-0790-4d42-9541-ade2388aacde-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c49eeabe-0790-4d42-9541-ade2388aacde-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:01.757488 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757495 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-trusted-ca\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757530 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2lbq\" (UniqueName: \"kubernetes.io/projected/c49eeabe-0790-4d42-9541-ade2388aacde-kube-api-access-z2lbq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:01.757553 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls podName:13bcb098-4a30-4e5f-8efe-ee7154bdee4a nodeName:}" failed. No retries permitted until 2026-09-11 13:54:02.257535371 +0000 UTC m=+59.127266617 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g2t9s" (UID: "13bcb098-4a30-4e5f-8efe-ee7154bdee4a") : secret "samples-operator-tls" not found Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757591 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-ca-trust-extracted\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-certificates\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5b6k\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-kube-api-access-c5b6k\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757688 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-installation-pull-secrets\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.757778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.757715 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-config\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.758287 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.758022 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c49eeabe-0790-4d42-9541-ade2388aacde-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.758287 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.758045 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-ca-trust-extracted\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.758561 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.758543 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-trusted-ca\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.758608 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.758595 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-certificates\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.759874 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.759854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-image-registry-private-configuration\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.760502 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.760473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c49eeabe-0790-4d42-9541-ade2388aacde-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.760585 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.760570 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-installation-pull-secrets\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.768678 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.768650 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5b6k\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-kube-api-access-c5b6k\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.768941 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.768916 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2lbq\" (UniqueName: \"kubernetes.io/projected/c49eeabe-0790-4d42-9541-ade2388aacde-kube-api-access-z2lbq\") pod \"kube-storage-version-migrator-operator-6c8f84f897-7k7gv\" (UID: \"c49eeabe-0790-4d42-9541-ade2388aacde\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.769652 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.769631 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj2w6\" (UniqueName: \"kubernetes.io/projected/10d8c0d5-cd9a-4ff7-a719-ea1ac937a986-kube-api-access-dj2w6\") pod \"volume-data-source-validator-86695b7cc7-ljsn4\" (UID: \"10d8c0d5-cd9a-4ff7-a719-ea1ac937a986\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" Sep 11 13:54:01.769766 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.769751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pt6s\" (UniqueName: \"kubernetes.io/projected/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-kube-api-access-8pt6s\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:01.772340 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.772324 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-bound-sa-token\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:01.836734 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.836650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:01.858773 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.858755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-config\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.858848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.858808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-serving-cert\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.858848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.858826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxfkw\" (UniqueName: \"kubernetes.io/projected/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-kube-api-access-dxfkw\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.859289 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.859265 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-config\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.860815 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.860800 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-serving-cert\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.872280 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.872263 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxfkw\" (UniqueName: \"kubernetes.io/projected/9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb-kube-api-access-dxfkw\") pod \"service-ca-operator-cf7c776b7-sh4lj\" (UID: \"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb\") " pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:01.873144 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.873090 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" Sep 11 13:54:01.917469 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.917436 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-mx52h"] Sep 11 13:54:01.920536 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:01.920510 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb22cf3e5_3c45_414e_8e59_22b25708f356.slice/crio-87dc46a7ab32fbbc344de1780a269a179a26f00af2b74f6042433e7ffe087f8e WatchSource:0}: Error finding container 87dc46a7ab32fbbc344de1780a269a179a26f00af2b74f6042433e7ffe087f8e: Status 404 returned error can't find the container with id 87dc46a7ab32fbbc344de1780a269a179a26f00af2b74f6042433e7ffe087f8e Sep 11 13:54:01.948600 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.948571 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" Sep 11 13:54:01.999820 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:01.999781 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" event={"ID":"b22cf3e5-3c45-414e-8e59-22b25708f356","Type":"ContainerStarted","Data":"87dc46a7ab32fbbc344de1780a269a179a26f00af2b74f6042433e7ffe087f8e"} Sep 11 13:54:02.041572 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:02.041548 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" Sep 11 13:54:02.184200 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:02.184178 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4"] Sep 11 13:54:02.185599 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:02.185570 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d8c0d5_cd9a_4ff7_a719_ea1ac937a986.slice/crio-e9a2316d515e7557e328138491aa855f247da120017e2dd10df364f90bc7c8c1 WatchSource:0}: Error finding container e9a2316d515e7557e328138491aa855f247da120017e2dd10df364f90bc7c8c1: Status 404 returned error can't find the container with id e9a2316d515e7557e328138491aa855f247da120017e2dd10df364f90bc7c8c1 Sep 11 13:54:02.242137 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:02.242092 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv"] Sep 11 13:54:02.245010 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:02.244991 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc49eeabe_0790_4d42_9541_ade2388aacde.slice/crio-9a881c8ca6ee72034fd43efda303e3621cb510b29231a524fe4689cdd81d8fe4 WatchSource:0}: Error finding container 9a881c8ca6ee72034fd43efda303e3621cb510b29231a524fe4689cdd81d8fe4: Status 404 returned error can't find the container with id 9a881c8ca6ee72034fd43efda303e3621cb510b29231a524fe4689cdd81d8fe4 Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:02.263409 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:02.263475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:02.263633 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:02.263693 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls podName:13bcb098-4a30-4e5f-8efe-ee7154bdee4a nodeName:}" failed. No retries permitted until 2026-09-11 13:54:03.263676239 +0000 UTC m=+60.133407496 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g2t9s" (UID: "13bcb098-4a30-4e5f-8efe-ee7154bdee4a") : secret "samples-operator-tls" not found Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:02.264173 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:02.264188 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f54f4c8f9-bk2d7: secret "image-registry-tls" not found Sep 11 13:54:02.266236 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:02.264240 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls podName:ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:03.264223761 +0000 UTC m=+60.133955016 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls") pod "image-registry-f54f4c8f9-bk2d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7") : secret "image-registry-tls" not found Sep 11 13:54:02.353257 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:02.353195 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj"] Sep 11 13:54:02.356780 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:02.356760 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0ddc27_1cbd_40ce_afcf_1a6967dedfdb.slice/crio-1f1757503c1002873c7c09c1589e4a8fe4e0e5ec2402c18d1e4b116071c64a51 WatchSource:0}: Error finding container 1f1757503c1002873c7c09c1589e4a8fe4e0e5ec2402c18d1e4b116071c64a51: Status 404 returned error can't find the container with id 1f1757503c1002873c7c09c1589e4a8fe4e0e5ec2402c18d1e4b116071c64a51 Sep 11 13:54:03.004233 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:03.004132 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" event={"ID":"c49eeabe-0790-4d42-9541-ade2388aacde","Type":"ContainerStarted","Data":"9a881c8ca6ee72034fd43efda303e3621cb510b29231a524fe4689cdd81d8fe4"} Sep 11 13:54:03.005805 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:03.005756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" event={"ID":"10d8c0d5-cd9a-4ff7-a719-ea1ac937a986","Type":"ContainerStarted","Data":"e9a2316d515e7557e328138491aa855f247da120017e2dd10df364f90bc7c8c1"} Sep 11 13:54:03.008633 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:03.008601 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" event={"ID":"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb","Type":"ContainerStarted","Data":"1f1757503c1002873c7c09c1589e4a8fe4e0e5ec2402c18d1e4b116071c64a51"} Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:03.272770 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:03.272849 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:03.272991 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:03.273005 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f54f4c8f9-bk2d7: secret "image-registry-tls" not found Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:03.273062 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls podName:ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:05.273045349 +0000 UTC m=+62.142776610 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls") pod "image-registry-f54f4c8f9-bk2d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7") : secret "image-registry-tls" not found Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:03.273186 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 13:54:03.273274 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:03.273240 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls podName:13bcb098-4a30-4e5f-8efe-ee7154bdee4a nodeName:}" failed. No retries permitted until 2026-09-11 13:54:05.27322621 +0000 UTC m=+62.142957457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g2t9s" (UID: "13bcb098-4a30-4e5f-8efe-ee7154bdee4a") : secret "samples-operator-tls" not found Sep 11 13:54:03.934984 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:03.934903 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w4pl7" Sep 11 13:54:04.014380 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:04.014341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" event={"ID":"10d8c0d5-cd9a-4ff7-a719-ea1ac937a986","Type":"ContainerStarted","Data":"02fe1547faf1cef1104ade9c552a83769fe7b17ecde42aec6e3044babf32eb80"} Sep 11 13:54:04.031711 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:04.031664 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-86695b7cc7-ljsn4" podStartSLOduration=1.5324528339999999 podStartE2EDuration="3.031646451s" totalImagesPullingTime="1.499193617s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:04.031166369 +0000 UTC m=+60.900897632" watchObservedRunningTime="2026-09-11 13:54:04.031646451 +0000 UTC m=+60.901377711" Sep 11 13:54:04.152564 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:04.152530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:04.251964 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:04.251837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:05.293542 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:05.293503 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:05.294039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:05.293635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:05.294039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:05.293670 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 13:54:05.294039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:05.293742 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:54:05.294039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:05.293750 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls podName:13bcb098-4a30-4e5f-8efe-ee7154bdee4a nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.293730846 +0000 UTC m=+66.163462102 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g2t9s" (UID: "13bcb098-4a30-4e5f-8efe-ee7154bdee4a") : secret "samples-operator-tls" not found Sep 11 13:54:05.294039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:05.293754 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f54f4c8f9-bk2d7: secret "image-registry-tls" not found Sep 11 13:54:05.294039 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:05.293787 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls podName:ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.29377609 +0000 UTC m=+66.163507343 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls") pod "image-registry-f54f4c8f9-bk2d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7") : secret "image-registry-tls" not found Sep 11 13:54:06.854454 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:06.854421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:06.931335 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:06.931297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:07.022695 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.022657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" event={"ID":"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb","Type":"ContainerStarted","Data":"8d32b49ca792085ff481f2a522efb5eeaf2775af9a4c650c7e0f2a84d6be288a"} Sep 11 13:54:07.024146 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.024114 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" event={"ID":"c49eeabe-0790-4d42-9541-ade2388aacde","Type":"ContainerStarted","Data":"de78fb0d0262c1cdd05ced3d7ededb7a93a0e3a61b4f88846aa867cc638a759f"} Sep 11 13:54:07.025622 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.025605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" event={"ID":"b22cf3e5-3c45-414e-8e59-22b25708f356","Type":"ContainerStarted","Data":"cbd9d4f1b165a8235e260f29c77a877329c174fecb2e5af13b7350fe6e4967e4"} Sep 11 13:54:07.039526 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.039482 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" podStartSLOduration=2.37939767 podStartE2EDuration="6.039469891s" totalImagesPullingTime="3.660072221s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:07.038874678 +0000 UTC m=+63.908605945" watchObservedRunningTime="2026-09-11 13:54:07.039469891 +0000 UTC m=+63.909201150" Sep 11 13:54:07.055654 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.055615 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-mx52h" podStartSLOduration=1.965457506 podStartE2EDuration="6.055601668s" totalImagesPullingTime="4.090144162s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:07.054661575 +0000 UTC m=+63.924392849" watchObservedRunningTime="2026-09-11 13:54:07.055601668 +0000 UTC m=+63.925332930" Sep 11 13:54:07.072510 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.072457 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" podStartSLOduration=2.301706321 podStartE2EDuration="6.072448622s" totalImagesPullingTime="3.770742301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:07.071609874 +0000 UTC m=+63.941341135" watchObservedRunningTime="2026-09-11 13:54:07.072448622 +0000 UTC m=+63.942179893" Sep 11 13:54:07.511541 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.511487 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:07.655616 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.655576 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:07.745228 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:07.745199 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:08.607752 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.607713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:08.634636 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.634608 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-s2z2k"] Sep 11 13:54:08.637182 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.637166 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.639815 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.639790 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 11 13:54:08.639946 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.639867 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-599v7\"" Sep 11 13:54:08.640066 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.640054 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 11 13:54:08.640124 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.640083 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 11 13:54:08.640823 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.640800 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 11 13:54:08.652998 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.652974 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2z2k"] Sep 11 13:54:08.718951 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.718924 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:08.723927 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.723902 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s9nf\" (UniqueName: \"kubernetes.io/projected/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-api-access-9s9nf\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.724027 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.723943 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-crio-socket\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.724078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.724033 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.724142 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.724074 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.724142 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.724134 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-data-volume\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825072 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s9nf\" (UniqueName: \"kubernetes.io/projected/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-api-access-9s9nf\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825191 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-crio-socket\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825191 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825171 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825304 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825304 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825235 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-crio-socket\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825304 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-data-volume\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.825304 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:08.825299 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:54:08.825491 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:08.825335 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:54:08.825491 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:08.825357 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.325340136 +0000 UTC m=+66.195071378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : configmap "kube-rbac-proxy" not found Sep 11 13:54:08.825491 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:08.825384 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:09.325372189 +0000 UTC m=+66.195103429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : secret "insights-runtime-extractor-tls" not found Sep 11 13:54:08.825654 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.825596 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-data-volume\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:08.834019 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:08.834000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s9nf\" (UniqueName: \"kubernetes.io/projected/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-api-access-9s9nf\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:09.210448 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.210413 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:09.285315 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.285284 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:09.330612 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.330589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:09.330747 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.330659 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:09.330747 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.330677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:09.330747 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.330694 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330751 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330769 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-f54f4c8f9-bk2d7: secret "image-registry-tls" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330771 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330793 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330816 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330834 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:10.330816905 +0000 UTC m=+67.200548161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : configmap "kube-rbac-proxy" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330850 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls podName:ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:17.330842652 +0000 UTC m=+74.200573892 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls") pod "image-registry-f54f4c8f9-bk2d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7") : secret "image-registry-tls" not found Sep 11 13:54:09.330862 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330863 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:10.330856269 +0000 UTC m=+67.200587509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : secret "insights-runtime-extractor-tls" not found Sep 11 13:54:09.331157 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.330874 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls podName:13bcb098-4a30-4e5f-8efe-ee7154bdee4a nodeName:}" failed. No retries permitted until 2026-09-11 13:54:17.330869567 +0000 UTC m=+74.200600807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-g2t9s" (UID: "13bcb098-4a30-4e5f-8efe-ee7154bdee4a") : secret "samples-operator-tls" not found Sep 11 13:54:09.431121 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.431081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:54:09.431213 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.431165 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:54:09.433650 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.433636 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 11 13:54:09.433808 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.433794 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 11 13:54:09.441549 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.441531 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 11 13:54:09.441607 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.441587 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs podName:8d081982-c02f-4f76-a910-bd0406b66e19 nodeName:}" failed. No retries permitted until 2026-09-11 13:55:13.441570795 +0000 UTC m=+130.311302035 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs") pod "network-metrics-daemon-zbh64" (UID: "8d081982-c02f-4f76-a910-bd0406b66e19") : secret "metrics-daemon-secret" not found Sep 11 13:54:09.443763 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.443749 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5b3eaae6-ad54-476c-8449-f8b9ab33a196-original-pull-secret\") pod \"global-pull-secret-syncer-nxrhp\" (UID: \"5b3eaae6-ad54-476c-8449-f8b9ab33a196\") " pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:54:09.532131 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.532050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:54:09.534563 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.534537 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxp5x\" (UniqueName: \"kubernetes.io/projected/7515a898-fe35-45cc-a38f-e1e14009910c-kube-api-access-fxp5x\") pod \"network-check-target-wl6rx\" (UID: \"7515a898-fe35-45cc-a38f-e1e14009910c\") " pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:54:09.556594 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.556576 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-nxrhp" Sep 11 13:54:09.566579 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.566561 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-spk9p\"" Sep 11 13:54:09.573405 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.573389 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:54:09.825968 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.825940 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-nxrhp"] Sep 11 13:54:09.830222 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:09.830201 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3eaae6_ad54_476c_8449_f8b9ab33a196.slice/crio-2d8f851ca512e901b21528c00a10ae07a71d91b1951b1fd52bbe4df8f093d6ab WatchSource:0}: Error finding container 2d8f851ca512e901b21528c00a10ae07a71d91b1951b1fd52bbe4df8f093d6ab: Status 404 returned error can't find the container with id 2d8f851ca512e901b21528c00a10ae07a71d91b1951b1fd52bbe4df8f093d6ab Sep 11 13:54:09.834816 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.834793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:54:09.834913 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.834894 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 11 13:54:09.834966 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:09.834941 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls podName:429e5452-0512-4330-8cf1-4581154d1055 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:41.834927464 +0000 UTC m=+98.704658708 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls") pod "dns-default-g8b9r" (UID: "429e5452-0512-4330-8cf1-4581154d1055") : secret "dns-default-metrics-tls" not found Sep 11 13:54:09.856414 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:09.856394 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wl6rx"] Sep 11 13:54:09.859054 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:09.859031 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7515a898_fe35_45cc_a38f_e1e14009910c.slice/crio-1fdc31b68794784136f777b0a1921856c5875f29843946a3828f324ca6ed5845 WatchSource:0}: Error finding container 1fdc31b68794784136f777b0a1921856c5875f29843946a3828f324ca6ed5845: Status 404 returned error can't find the container with id 1fdc31b68794784136f777b0a1921856c5875f29843946a3828f324ca6ed5845 Sep 11 13:54:10.035995 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.035959 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wl6rx" event={"ID":"7515a898-fe35-45cc-a38f-e1e14009910c","Type":"ContainerStarted","Data":"02961c1d73b6755c4b829cdf1d1b5cc1e1e776617a219af55d395aecf67b65f4"} Sep 11 13:54:10.036191 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.036021 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:54:10.036191 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.036040 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wl6rx" event={"ID":"7515a898-fe35-45cc-a38f-e1e14009910c","Type":"ContainerStarted","Data":"1fdc31b68794784136f777b0a1921856c5875f29843946a3828f324ca6ed5845"} Sep 11 13:54:10.037261 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.037237 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nxrhp" event={"ID":"5b3eaae6-ad54-476c-8449-f8b9ab33a196","Type":"ContainerStarted","Data":"2d8f851ca512e901b21528c00a10ae07a71d91b1951b1fd52bbe4df8f093d6ab"} Sep 11 13:54:10.059224 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.059157 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wl6rx" podStartSLOduration=66.059140938 podStartE2EDuration="1m6.059140938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:10.058927098 +0000 UTC m=+66.928658360" watchObservedRunningTime="2026-09-11 13:54:10.059140938 +0000 UTC m=+66.928872200" Sep 11 13:54:10.338059 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.338030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:10.338202 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.338073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:10.338266 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:10.338199 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:54:10.338266 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:10.338200 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:54:10.338266 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:10.338262 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:12.338248358 +0000 UTC m=+69.207979603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : secret "insights-runtime-extractor-tls" not found Sep 11 13:54:10.338394 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:10.338297 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:12.338281033 +0000 UTC m=+69.208012287 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : configmap "kube-rbac-proxy" not found Sep 11 13:54:10.449630 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:10.449596 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:11.537677 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:11.537651 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kkhl9_06d275ee-fd61-4460-8bb1-013e6f6d08d2/dns-node-resolver/0.log" Sep 11 13:54:12.335271 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:12.335240 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jttfb_9036a4b6-21fd-47ea-9092-556af9ba0a7b/node-ca/0.log" Sep 11 13:54:12.355090 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:12.355063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:12.355255 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:12.355134 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:12.355255 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:12.355222 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:54:12.355255 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:12.355243 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:54:12.355408 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:12.355315 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:16.355295763 +0000 UTC m=+73.225027020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : secret "insights-runtime-extractor-tls" not found Sep 11 13:54:12.355408 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:12.355331 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:16.355325336 +0000 UTC m=+73.225056581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : configmap "kube-rbac-proxy" not found Sep 11 13:54:13.135461 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.135428 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jgljh_596d001e-42a1-409b-95dd-71995ca53f2e/serve-healthcheck-canary/0.log" Sep 11 13:54:13.378379 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.378347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:13.523159 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.523058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:13.624831 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.624798 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:13.742717 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.742688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:13.849038 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.849000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:13.939416 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.939389 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-7k7gv_c49eeabe-0790-4d42-9541-ade2388aacde/kube-storage-version-migrator-operator/0.log" Sep 11 13:54:13.954087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:13.954052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:14.053214 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:14.053188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-nxrhp" event={"ID":"5b3eaae6-ad54-476c-8449-f8b9ab33a196","Type":"ContainerStarted","Data":"80f9dbc178abe3ed1de1b58d31f44be00a13b7e4e00630a7e7921f9cc1c7488b"} Sep 11 13:54:14.131257 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:14.131194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:14.231254 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:14.231226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:15.261408 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:15.261377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:15.450114 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:15.450081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:15.671812 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:15.671788 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:15.846011 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:15.845979 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:16.049018 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:16.048993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:16.388972 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:16.388867 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:16.388972 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:16.388942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:16.389420 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:16.389024 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 11 13:54:16.389420 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:16.389070 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 11 13:54:16.389420 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:16.389087 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:24.389072213 +0000 UTC m=+81.258803453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : secret "insights-runtime-extractor-tls" not found Sep 11 13:54:16.389420 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:16.389135 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm podName:e3b0b9eb-9b0c-4839-a125-edeb7922bc51 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:24.38912304 +0000 UTC m=+81.258854279 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm") pod "insights-runtime-extractor-s2z2k" (UID: "e3b0b9eb-9b0c-4839-a125-edeb7922bc51") : configmap "kube-rbac-proxy" not found Sep 11 13:54:16.952630 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:16.952595 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:17.283996 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.283916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:17.377463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.377428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:17.396465 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.396430 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:17.396794 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.396493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:17.398947 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.398927 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"image-registry-f54f4c8f9-bk2d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:17.399007 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.398969 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/13bcb098-4a30-4e5f-8efe-ee7154bdee4a-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-g2t9s\" (UID: \"13bcb098-4a30-4e5f-8efe-ee7154bdee4a\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:17.480545 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.480508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:17.497428 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.497410 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-jpj5q\"" Sep 11 13:54:17.505402 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.505382 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" Sep 11 13:54:17.519900 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.519877 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bjrq5\"" Sep 11 13:54:17.528161 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.528133 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:17.657780 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.657748 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:17.783507 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.783462 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-nxrhp" podStartSLOduration=69.971063361 podStartE2EDuration="1m13.783446061s" totalImagesPullingTime="3.8123827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:14.110964255 +0000 UTC m=+70.980695516" watchObservedRunningTime="2026-09-11 13:54:17.783446061 +0000 UTC m=+74.653177323" Sep 11 13:54:17.784047 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.784032 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s"] Sep 11 13:54:17.811951 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.811932 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-f54f4c8f9-bk2d7"] Sep 11 13:54:17.814475 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:17.814452 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba4a84c6_9cfe_41d0_bbfc_2d8236d687d7.slice/crio-910e2527eea671720294ed022c9ffba0f6478c43d8b19f0884742cc7b7c81a17 WatchSource:0}: Error finding container 910e2527eea671720294ed022c9ffba0f6478c43d8b19f0884742cc7b7c81a17: Status 404 returned error can't find the container with id 910e2527eea671720294ed022c9ffba0f6478c43d8b19f0884742cc7b7c81a17 Sep 11 13:54:17.894791 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:17.894750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:18.019977 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.019940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:18.068998 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.068965 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" event={"ID":"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7","Type":"ContainerStarted","Data":"373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c"} Sep 11 13:54:18.069167 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.069022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" event={"ID":"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7","Type":"ContainerStarted","Data":"910e2527eea671720294ed022c9ffba0f6478c43d8b19f0884742cc7b7c81a17"} Sep 11 13:54:18.069167 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.069062 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:18.070076 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.070049 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" event={"ID":"13bcb098-4a30-4e5f-8efe-ee7154bdee4a","Type":"ContainerStarted","Data":"c865ab8bd4834cf29a3ad9eb3e486c1306078071f7b87a9ebd4af45ca472a72c"} Sep 11 13:54:18.101995 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.101955 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" podStartSLOduration=17.101940168 podStartE2EDuration="17.101940168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:18.101213899 +0000 UTC m=+74.970945156" watchObservedRunningTime="2026-09-11 13:54:18.101940168 +0000 UTC m=+74.971671433" Sep 11 13:54:18.109737 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:18.109712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:19.053483 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:19.053445 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:19.312699 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:19.312629 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:19.781034 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:19.781005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:19.881943 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:19.881862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:20.071798 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:20.071764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:20.077694 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:20.077672 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" event={"ID":"13bcb098-4a30-4e5f-8efe-ee7154bdee4a","Type":"ContainerStarted","Data":"b54b138557184c1aa0493cd878dc0e02cee8fec66ffa39ec6f95a098124b262c"} Sep 11 13:54:20.077798 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:20.077729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" event={"ID":"13bcb098-4a30-4e5f-8efe-ee7154bdee4a","Type":"ContainerStarted","Data":"04bcdb6793e7530a8ee8b00d1ffce5b15f064068c8f458ed9b4bde9a69996651"} Sep 11 13:54:20.098397 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:20.098357 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-g2t9s" podStartSLOduration=17.312347046 podStartE2EDuration="19.098343872s" totalImagesPullingTime="1.785996826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:20.097562773 +0000 UTC m=+76.967294036" watchObservedRunningTime="2026-09-11 13:54:20.098343872 +0000 UTC m=+76.968075133" Sep 11 13:54:20.147541 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:20.147482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:20.859681 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:20.859644 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:21.468865 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:21.468830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:21.652235 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:21.652197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:21.939283 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:21.939246 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:22.048396 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:22.048365 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:22.269253 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:22.269166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:22.662938 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:22.662911 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:23.260282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:23.260249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:23.675728 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:23.675705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:23.857764 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:23.857742 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:24.057999 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.057978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:24.451845 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.451773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:24.451845 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.451808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:24.452397 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.452380 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:24.454291 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.454276 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/e3b0b9eb-9b0c-4839-a125-edeb7922bc51-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-s2z2k\" (UID: \"e3b0b9eb-9b0c-4839-a125-edeb7922bc51\") " pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:24.547371 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.547348 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-s2z2k" Sep 11 13:54:24.821297 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:24.821152 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-s2z2k"] Sep 11 13:54:24.824237 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:24.824194 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b0b9eb_9b0c_4839_a125_edeb7922bc51.slice/crio-73abf761cf0620e1af88be6b2a7e8d0cbb6714f7febb32ed644e88f773efc972 WatchSource:0}: Error finding container 73abf761cf0620e1af88be6b2a7e8d0cbb6714f7febb32ed644e88f773efc972: Status 404 returned error can't find the container with id 73abf761cf0620e1af88be6b2a7e8d0cbb6714f7febb32ed644e88f773efc972 Sep 11 13:54:25.094531 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:25.094466 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2z2k" event={"ID":"e3b0b9eb-9b0c-4839-a125-edeb7922bc51","Type":"ContainerStarted","Data":"611d5b75cc2b7f64792dab6302755d7ac82ef8dc58bc5806889bf57301c66f33"} Sep 11 13:54:25.094531 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:25.094515 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2z2k" event={"ID":"e3b0b9eb-9b0c-4839-a125-edeb7922bc51","Type":"ContainerStarted","Data":"73abf761cf0620e1af88be6b2a7e8d0cbb6714f7febb32ed644e88f773efc972"} Sep 11 13:54:26.098803 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:26.098770 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2z2k" event={"ID":"e3b0b9eb-9b0c-4839-a125-edeb7922bc51","Type":"ContainerStarted","Data":"74be36d3b964b06b5f9051c5a784baee645777284a25f15a51019b293685c018"} Sep 11 13:54:28.106789 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:28.106755 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-s2z2k" event={"ID":"e3b0b9eb-9b0c-4839-a125-edeb7922bc51","Type":"ContainerStarted","Data":"70edec4b0cfba6918cf49887888103a8f06a79e853c3712b5c698c2236b1929c"} Sep 11 13:54:28.128872 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:28.128828 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-s2z2k" podStartSLOduration=17.96846501 podStartE2EDuration="20.128813311s" totalImagesPullingTime="2.160348301s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:54:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:28.128690653 +0000 UTC m=+84.998421916" watchObservedRunningTime="2026-09-11 13:54:28.128813311 +0000 UTC m=+84.998544603" Sep 11 13:54:31.041920 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:31.041888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:31.144710 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:31.144679 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:32.736577 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.736536 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz"] Sep 11 13:54:32.740810 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.740789 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.742309 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.742285 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f54f4c8f9-bk2d7"] Sep 11 13:54:32.743949 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.743915 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rlg6w\"" Sep 11 13:54:32.743949 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.743941 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 11 13:54:32.744143 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.744021 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 11 13:54:32.747468 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.747442 2579 patch_prober.go:28] interesting pod/image-registry-f54f4c8f9-bk2d7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 13:54:32.747573 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.747493 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 13:54:32.749306 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.749285 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz"] Sep 11 13:54:32.814001 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.813964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6bf66f8-5d35-44c4-ba30-4d52c5519a22-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4sfhz\" (UID: \"d6bf66f8-5d35-44c4-ba30-4d52c5519a22\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.814001 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.814004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bf66f8-5d35-44c4-ba30-4d52c5519a22-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sfhz\" (UID: \"d6bf66f8-5d35-44c4-ba30-4d52c5519a22\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.835415 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.835384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:32.841770 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.841750 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-797f7fc5b4-852mw"] Sep 11 13:54:32.844813 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.844797 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.866854 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.866833 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797f7fc5b4-852mw"] Sep 11 13:54:32.915215 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d18dcec8-7619-4a8a-b790-15564fb955dc-ca-trust-extracted\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915338 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915222 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6bf66f8-5d35-44c4-ba30-4d52c5519a22-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4sfhz\" (UID: \"d6bf66f8-5d35-44c4-ba30-4d52c5519a22\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.915338 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-registry-tls\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915338 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bf66f8-5d35-44c4-ba30-4d52c5519a22-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sfhz\" (UID: \"d6bf66f8-5d35-44c4-ba30-4d52c5519a22\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.915338 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d18dcec8-7619-4a8a-b790-15564fb955dc-trusted-ca\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915502 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d18dcec8-7619-4a8a-b790-15564fb955dc-installation-pull-secrets\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915502 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915394 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-bound-sa-token\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915578 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915504 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d18dcec8-7619-4a8a-b790-15564fb955dc-registry-certificates\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915578 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scnsp\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-kube-api-access-scnsp\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915659 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915577 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d18dcec8-7619-4a8a-b790-15564fb955dc-image-registry-private-configuration\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:32.915874 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.915859 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/d6bf66f8-5d35-44c4-ba30-4d52c5519a22-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-4sfhz\" (UID: \"d6bf66f8-5d35-44c4-ba30-4d52c5519a22\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.917794 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.917779 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/d6bf66f8-5d35-44c4-ba30-4d52c5519a22-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-4sfhz\" (UID: \"d6bf66f8-5d35-44c4-ba30-4d52c5519a22\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:32.929056 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:32.929034 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:33.016844 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.016791 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d18dcec8-7619-4a8a-b790-15564fb955dc-trusted-ca\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.016844 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.016823 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d18dcec8-7619-4a8a-b790-15564fb955dc-installation-pull-secrets\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.016957 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.016846 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-bound-sa-token\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017024 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017003 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d18dcec8-7619-4a8a-b790-15564fb955dc-registry-certificates\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017064 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scnsp\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-kube-api-access-scnsp\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017064 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d18dcec8-7619-4a8a-b790-15564fb955dc-image-registry-private-configuration\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017184 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d18dcec8-7619-4a8a-b790-15564fb955dc-ca-trust-extracted\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017235 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-registry-tls\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017553 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017527 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d18dcec8-7619-4a8a-b790-15564fb955dc-ca-trust-extracted\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017816 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d18dcec8-7619-4a8a-b790-15564fb955dc-trusted-ca\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.017914 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.017827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d18dcec8-7619-4a8a-b790-15564fb955dc-registry-certificates\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.019466 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.019442 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d18dcec8-7619-4a8a-b790-15564fb955dc-installation-pull-secrets\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.019885 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.019863 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d18dcec8-7619-4a8a-b790-15564fb955dc-image-registry-private-configuration\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.019944 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.019883 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-registry-tls\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.028220 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.028195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scnsp\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-kube-api-access-scnsp\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.028582 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.028551 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d18dcec8-7619-4a8a-b790-15564fb955dc-bound-sa-token\") pod \"image-registry-797f7fc5b4-852mw\" (UID: \"d18dcec8-7619-4a8a-b790-15564fb955dc\") " pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.052171 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.052151 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" Sep 11 13:54:33.153792 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.153770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:33.321625 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.321600 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz"] Sep 11 13:54:33.324184 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:33.324163 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bf66f8_5d35_44c4_ba30_4d52c5519a22.slice/crio-10e46b6eb97aacd496e97417941d9319575dbdb34712e8a220a8e77701fbbb5c WatchSource:0}: Error finding container 10e46b6eb97aacd496e97417941d9319575dbdb34712e8a220a8e77701fbbb5c: Status 404 returned error can't find the container with id 10e46b6eb97aacd496e97417941d9319575dbdb34712e8a220a8e77701fbbb5c Sep 11 13:54:33.429347 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:33.429328 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-797f7fc5b4-852mw"] Sep 11 13:54:33.431225 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:33.431203 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd18dcec8_7619_4a8a_b790_15564fb955dc.slice/crio-921101bb193d99bf14b3c74c5939ddae80354c1f1dae638c484428decb78538f WatchSource:0}: Error finding container 921101bb193d99bf14b3c74c5939ddae80354c1f1dae638c484428decb78538f: Status 404 returned error can't find the container with id 921101bb193d99bf14b3c74c5939ddae80354c1f1dae638c484428decb78538f Sep 11 13:54:34.128629 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:34.128593 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" event={"ID":"d18dcec8-7619-4a8a-b790-15564fb955dc","Type":"ContainerStarted","Data":"45ff0a78d7ff9bdea5c3b5402541e072d5891f0687e39e61ffaa8dbb7e2fa2df"} Sep 11 13:54:34.129034 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:34.128642 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" event={"ID":"d18dcec8-7619-4a8a-b790-15564fb955dc","Type":"ContainerStarted","Data":"921101bb193d99bf14b3c74c5939ddae80354c1f1dae638c484428decb78538f"} Sep 11 13:54:34.129034 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:34.128680 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:34.129855 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:34.129838 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" event={"ID":"d6bf66f8-5d35-44c4-ba30-4d52c5519a22","Type":"ContainerStarted","Data":"10e46b6eb97aacd496e97417941d9319575dbdb34712e8a220a8e77701fbbb5c"} Sep 11 13:54:34.166398 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:34.166357 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" podStartSLOduration=2.166344884 podStartE2EDuration="2.166344884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:34.165672399 +0000 UTC m=+91.035403662" watchObservedRunningTime="2026-09-11 13:54:34.166344884 +0000 UTC m=+91.036076146" Sep 11 13:54:35.134417 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:35.134379 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" event={"ID":"d6bf66f8-5d35-44c4-ba30-4d52c5519a22","Type":"ContainerStarted","Data":"4b714ae3286f6dcfc03244d3e3fbdc725eb019f9c4f9ce3378626bb0eb4be1ee"} Sep 11 13:54:35.150639 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:35.150596 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-4sfhz" podStartSLOduration=1.870044569 podStartE2EDuration="3.150580932s" totalImagesPullingTime="1.280536363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:35.149961439 +0000 UTC m=+92.019692701" watchObservedRunningTime="2026-09-11 13:54:35.150580932 +0000 UTC m=+92.020312195" Sep 11 13:54:35.299765 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:35.299736 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:35.409551 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:35.409471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:35.827087 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:35.827060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:35.928823 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:35.928794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:36.036037 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:36.036002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:36.451386 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:36.451329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:39.027387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.027354 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:39.065062 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.065031 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl"] Sep 11 13:54:39.096137 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.096089 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl"] Sep 11 13:54:39.096260 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.096189 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.099322 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.099296 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 11 13:54:39.099451 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.099407 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Sep 11 13:54:39.099513 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.099499 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 11 13:54:39.099565 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.099545 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 11 13:54:39.099623 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.099573 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Sep 11 13:54:39.099623 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.099617 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-x5llw\"" Sep 11 13:54:39.119960 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.119928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:39.193034 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.193002 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:39.262325 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.262296 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbll\" (UniqueName: \"kubernetes.io/projected/38cabd2b-39bc-4cfb-a243-ef957d6d825c-kube-api-access-6pbll\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.262438 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.262353 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38cabd2b-39bc-4cfb-a243-ef957d6d825c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.262483 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.262454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38cabd2b-39bc-4cfb-a243-ef957d6d825c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.262521 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.262492 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38cabd2b-39bc-4cfb-a243-ef957d6d825c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.362856 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.362827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38cabd2b-39bc-4cfb-a243-ef957d6d825c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.362967 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.362881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38cabd2b-39bc-4cfb-a243-ef957d6d825c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.362967 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.362906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38cabd2b-39bc-4cfb-a243-ef957d6d825c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.362967 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.362931 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbll\" (UniqueName: \"kubernetes.io/projected/38cabd2b-39bc-4cfb-a243-ef957d6d825c-kube-api-access-6pbll\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.363674 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.363602 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/38cabd2b-39bc-4cfb-a243-ef957d6d825c-metrics-client-ca\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.365339 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.365311 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/38cabd2b-39bc-4cfb-a243-ef957d6d825c-prometheus-operator-tls\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.365560 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.365541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/38cabd2b-39bc-4cfb-a243-ef957d6d825c-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.372291 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.372271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbll\" (UniqueName: \"kubernetes.io/projected/38cabd2b-39bc-4cfb-a243-ef957d6d825c-kube-api-access-6pbll\") pod \"prometheus-operator-68bcd7f987-zpcgl\" (UID: \"38cabd2b-39bc-4cfb-a243-ef957d6d825c\") " pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.406247 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.406222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" Sep 11 13:54:39.705201 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:39.705131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl"] Sep 11 13:54:39.708323 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:39.708299 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cabd2b_39bc_4cfb_a243_ef957d6d825c.slice/crio-4628ceda5984a4fe78635ed2885618266960b0cdc3a3a024e0228b482e12373b WatchSource:0}: Error finding container 4628ceda5984a4fe78635ed2885618266960b0cdc3a3a024e0228b482e12373b: Status 404 returned error can't find the container with id 4628ceda5984a4fe78635ed2885618266960b0cdc3a3a024e0228b482e12373b Sep 11 13:54:40.152632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:40.152599 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" event={"ID":"38cabd2b-39bc-4cfb-a243-ef957d6d825c","Type":"ContainerStarted","Data":"4628ceda5984a4fe78635ed2885618266960b0cdc3a3a024e0228b482e12373b"} Sep 11 13:54:41.044002 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:41.043976 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wl6rx" Sep 11 13:54:41.508669 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:41.508640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:41.881108 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:41.881076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:54:41.883510 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:41.883478 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/429e5452-0512-4330-8cf1-4581154d1055-metrics-tls\") pod \"dns-default-g8b9r\" (UID: \"429e5452-0512-4330-8cf1-4581154d1055\") " pod="openshift-dns/dns-default-g8b9r" Sep 11 13:54:41.923066 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:41.923044 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-nllbd\"" Sep 11 13:54:41.931417 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:41.931394 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-g8b9r" Sep 11 13:54:42.162089 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:42.162011 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" event={"ID":"38cabd2b-39bc-4cfb-a243-ef957d6d825c","Type":"ContainerStarted","Data":"46a2500d1268178adc8b8469086f242021968e743a556d216f6201f69f6b41f8"} Sep 11 13:54:42.162089 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:42.162061 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" event={"ID":"38cabd2b-39bc-4cfb-a243-ef957d6d825c","Type":"ContainerStarted","Data":"33ce60bd46c988ba7e685197ea59336ed7cb705720566d20a8740a96eeb9fd11"} Sep 11 13:54:42.179767 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:42.179726 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-68bcd7f987-zpcgl" podStartSLOduration=1.6693725270000002 podStartE2EDuration="3.179707673s" totalImagesPullingTime="1.510335146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:42.178910448 +0000 UTC m=+99.048641721" watchObservedRunningTime="2026-09-11 13:54:42.179707673 +0000 UTC m=+99.049438935" Sep 11 13:54:42.202181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:42.202159 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-g8b9r"] Sep 11 13:54:42.204965 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:42.204939 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod429e5452_0512_4330_8cf1_4581154d1055.slice/crio-7b49225266f8c9eacfcf7d194d231c1b5f6b55f669e19d8a676505f66d6f1bcd WatchSource:0}: Error finding container 7b49225266f8c9eacfcf7d194d231c1b5f6b55f669e19d8a676505f66d6f1bcd: Status 404 returned error can't find the container with id 7b49225266f8c9eacfcf7d194d231c1b5f6b55f669e19d8a676505f66d6f1bcd Sep 11 13:54:42.747868 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:42.747834 2579 patch_prober.go:28] interesting pod/image-registry-f54f4c8f9-bk2d7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 13:54:42.748276 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:42.747890 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 13:54:43.166811 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:43.166764 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g8b9r" event={"ID":"429e5452-0512-4330-8cf1-4581154d1055","Type":"ContainerStarted","Data":"7b49225266f8c9eacfcf7d194d231c1b5f6b55f669e19d8a676505f66d6f1bcd"} Sep 11 13:54:43.824785 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:43.824757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:43.928353 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:43.928317 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.172549 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.172469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g8b9r" event={"ID":"429e5452-0512-4330-8cf1-4581154d1055","Type":"ContainerStarted","Data":"bafd8a48d61c6b2c3f94a23a6a555d371cc926d98e78e75fa11a661dc1ebc16a"} Sep 11 13:54:44.172549 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.172521 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-g8b9r" event={"ID":"429e5452-0512-4330-8cf1-4581154d1055","Type":"ContainerStarted","Data":"e581f5b5c7e51b9b988d09fff80a872d19b768d1690b32c381628313acfcc7ef"} Sep 11 13:54:44.172781 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.172611 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-g8b9r" Sep 11 13:54:44.214209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.214165 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-g8b9r" podStartSLOduration=65.62033490499999 podStartE2EDuration="1m7.214151193s" totalImagesPullingTime="1.593816288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:44.212918825 +0000 UTC m=+101.082650087" watchObservedRunningTime="2026-09-11 13:54:44.214151193 +0000 UTC m=+101.083882449" Sep 11 13:54:44.218686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.218662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.326260 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.326227 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.524370 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.524248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.563691 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.563662 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8"] Sep 11 13:54:44.565826 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.565812 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.568935 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.568920 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sbf62\"" Sep 11 13:54:44.569847 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.569830 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Sep 11 13:54:44.569929 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.569835 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Sep 11 13:54:44.569929 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.569835 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Sep 11 13:54:44.579712 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.579692 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8"] Sep 11 13:54:44.582540 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.582523 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vk9hj"] Sep 11 13:54:44.584457 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.584439 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.590357 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.590333 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 11 13:54:44.590592 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.590577 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 11 13:54:44.590725 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.590705 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 11 13:54:44.590855 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.590841 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6tn2c\"" Sep 11 13:54:44.628863 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.628834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.708323 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a33cad5-3e43-4d5f-9df0-227a64c286b3-metrics-client-ca\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708323 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-sys\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708488 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708488 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708488 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708465 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-textfile\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708583 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.708583 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2bc881d3-aa82-48b9-a847-dc3c171d514c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.708583 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.708680 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708582 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42wnd\" (UniqueName: \"kubernetes.io/projected/9a33cad5-3e43-4d5f-9df0-227a64c286b3-kube-api-access-42wnd\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708680 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-wtmp\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708680 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708656 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.708769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708707 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj86x\" (UniqueName: \"kubernetes.io/projected/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-api-access-dj86x\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.708769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708730 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.708769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc881d3-aa82-48b9-a847-dc3c171d514c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.708859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.708782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-root\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.730755 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.730729 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.809634 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a33cad5-3e43-4d5f-9df0-227a64c286b3-metrics-client-ca\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.809740 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-sys\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.809740 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.809740 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809729 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-sys\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.809890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.809890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-textfile\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.809890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809833 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.809890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2bc881d3-aa82-48b9-a847-dc3c171d514c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.810082 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.810082 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42wnd\" (UniqueName: \"kubernetes.io/projected/9a33cad5-3e43-4d5f-9df0-227a64c286b3-kube-api-access-42wnd\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810082 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.809983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-wtmp\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810082 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.810082 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj86x\" (UniqueName: \"kubernetes.io/projected/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-api-access-dj86x\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810083 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc881d3-aa82-48b9-a847-dc3c171d514c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810160 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-textfile\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810167 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-root\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810284 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-accelerators-collector-config\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:44.810289 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/2bc881d3-aa82-48b9-a847-dc3c171d514c-volume-directive-shadow\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810320 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9a33cad5-3e43-4d5f-9df0-227a64c286b3-metrics-client-ca\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810365 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:44.810362 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls podName:9a33cad5-3e43-4d5f-9df0-227a64c286b3 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:45.310344874 +0000 UTC m=+102.180076128 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls") pod "node-exporter-vk9hj" (UID: "9a33cad5-3e43-4d5f-9df0-227a64c286b3") : secret "node-exporter-tls" not found Sep 11 13:54:44.810829 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-root\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810829 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-wtmp\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.810829 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810816 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2bc881d3-aa82-48b9-a847-dc3c171d514c-metrics-client-ca\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.811179 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.810856 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.812625 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.812600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.812755 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.812737 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-tls\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.812816 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.812770 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.820561 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.820541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj86x\" (UniqueName: \"kubernetes.io/projected/2bc881d3-aa82-48b9-a847-dc3c171d514c-kube-api-access-dj86x\") pod \"kube-state-metrics-59fbdd77d-7ljt8\" (UID: \"2bc881d3-aa82-48b9-a847-dc3c171d514c\") " pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.823038 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.823017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42wnd\" (UniqueName: \"kubernetes.io/projected/9a33cad5-3e43-4d5f-9df0-227a64c286b3-kube-api-access-42wnd\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:44.832379 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.832358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:44.874829 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.874811 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" Sep 11 13:54:44.942659 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:44.942624 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:45.152224 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:45.152201 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8"] Sep 11 13:54:45.154611 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:45.154582 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bc881d3_aa82_48b9_a847_dc3c171d514c.slice/crio-fdca26f33e42c4bd0fd20817c5b48d71dc63efd68af6488865e74e9d0906c48e WatchSource:0}: Error finding container fdca26f33e42c4bd0fd20817c5b48d71dc63efd68af6488865e74e9d0906c48e: Status 404 returned error can't find the container with id fdca26f33e42c4bd0fd20817c5b48d71dc63efd68af6488865e74e9d0906c48e Sep 11 13:54:45.177057 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:45.177034 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" event={"ID":"2bc881d3-aa82-48b9-a847-dc3c171d514c","Type":"ContainerStarted","Data":"fdca26f33e42c4bd0fd20817c5b48d71dc63efd68af6488865e74e9d0906c48e"} Sep 11 13:54:45.314873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:45.314843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:45.315036 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:45.314959 2579 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 11 13:54:45.315036 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:45.315013 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls podName:9a33cad5-3e43-4d5f-9df0-227a64c286b3 nodeName:}" failed. No retries permitted until 2026-09-11 13:54:46.314998961 +0000 UTC m=+103.184730204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls") pod "node-exporter-vk9hj" (UID: "9a33cad5-3e43-4d5f-9df0-227a64c286b3") : secret "node-exporter-tls" not found Sep 11 13:54:45.369569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:45.369544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:45.503178 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:45.503090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:45.680294 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:45.680239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:46.039449 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:46.039411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:46.324253 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:46.324223 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:46.326756 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:46.326735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9a33cad5-3e43-4d5f-9df0-227a64c286b3-node-exporter-tls\") pod \"node-exporter-vk9hj\" (UID: \"9a33cad5-3e43-4d5f-9df0-227a64c286b3\") " pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:46.392628 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:46.392600 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vk9hj" Sep 11 13:54:46.416509 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:46.416480 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a33cad5_3e43_4d5f_9df0_227a64c286b3.slice/crio-99a828f1f8d573e9a8a52562dff5412aa4252bb7a7f784c881cce9851953bc5c WatchSource:0}: Error finding container 99a828f1f8d573e9a8a52562dff5412aa4252bb7a7f784c881cce9851953bc5c: Status 404 returned error can't find the container with id 99a828f1f8d573e9a8a52562dff5412aa4252bb7a7f784c881cce9851953bc5c Sep 11 13:54:46.445735 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:46.445712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:47.186959 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.186868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vk9hj" event={"ID":"9a33cad5-3e43-4d5f-9df0-227a64c286b3","Type":"ContainerStarted","Data":"99a828f1f8d573e9a8a52562dff5412aa4252bb7a7f784c881cce9851953bc5c"} Sep 11 13:54:47.189889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.189861 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" event={"ID":"2bc881d3-aa82-48b9-a847-dc3c171d514c","Type":"ContainerStarted","Data":"c9306a89c4c6ff20c11a0f18ac50f5933b05cd805379290790cb4523a438b17d"} Sep 11 13:54:47.190023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.189920 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" event={"ID":"2bc881d3-aa82-48b9-a847-dc3c171d514c","Type":"ContainerStarted","Data":"7b85702b9044528462fc1166344a6321233b2aee2f1d20bdd29ab26deb8eeb26"} Sep 11 13:54:47.190023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.189947 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" event={"ID":"2bc881d3-aa82-48b9-a847-dc3c171d514c","Type":"ContainerStarted","Data":"2a75d44bba3ba1adf36fd0f9139f290f6c4dcc06f1ae21a714f33350cd79897c"} Sep 11 13:54:47.236045 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.235997 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-59fbdd77d-7ljt8" podStartSLOduration=1.716435962 podStartE2EDuration="3.235981599s" totalImagesPullingTime="1.519545637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:47.233885944 +0000 UTC m=+104.103617202" watchObservedRunningTime="2026-09-11 13:54:47.235981599 +0000 UTC m=+104.105712861" Sep 11 13:54:47.841875 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.841786 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:47.941007 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:47.940971 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:48.153792 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:48.153702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:48.194767 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:48.194737 2579 generic.go:356] "Generic (PLEG): container finished" podID="9a33cad5-3e43-4d5f-9df0-227a64c286b3" containerID="76901099d0dad290ad9454bf80f0e6b73899ae01a4be64d9bfacb2265701236f" exitCode=0 Sep 11 13:54:48.195185 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:48.194811 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vk9hj" event={"ID":"9a33cad5-3e43-4d5f-9df0-227a64c286b3","Type":"ContainerDied","Data":"76901099d0dad290ad9454bf80f0e6b73899ae01a4be64d9bfacb2265701236f"} Sep 11 13:54:48.377675 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:48.377638 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:49.201217 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.201178 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vk9hj" event={"ID":"9a33cad5-3e43-4d5f-9df0-227a64c286b3","Type":"ContainerStarted","Data":"1ad690c4d444576a3662e9a5b585a4cf94204eb4257ee77e1cf8fe1e63b5dbb7"} Sep 11 13:54:49.201604 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.201239 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vk9hj" event={"ID":"9a33cad5-3e43-4d5f-9df0-227a64c286b3","Type":"ContainerStarted","Data":"2c598e75ac8e8479281ed9a3bb6f9773c179780aebf8df3a46fc6aa356167c54"} Sep 11 13:54:49.240619 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.240446 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vk9hj" podStartSLOduration=4.00647653 podStartE2EDuration="5.24042629s" totalImagesPullingTime="1.163374734s" totalInitContainerRuntime="70.575026ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:54:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:49.238716841 +0000 UTC m=+106.108448094" watchObservedRunningTime="2026-09-11 13:54:49.24042629 +0000 UTC m=+106.110157557" Sep 11 13:54:49.241815 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.241792 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:49.261524 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.261498 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9"] Sep 11 13:54:49.263304 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.263290 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.266078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.266062 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-xgbrf\"" Sep 11 13:54:49.266319 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.266306 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Sep 11 13:54:49.278564 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.278542 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9"] Sep 11 13:54:49.450887 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.450860 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8xfv\" (UniqueName: \"kubernetes.io/projected/95aff6cc-a715-473c-8e92-9303dc3539b4-kube-api-access-b8xfv\") pod \"monitoring-plugin-5d85778b9d-8kfw9\" (UID: \"95aff6cc-a715-473c-8e92-9303dc3539b4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.451040 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.450914 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/95aff6cc-a715-473c-8e92-9303dc3539b4-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kfw9\" (UID: \"95aff6cc-a715-473c-8e92-9303dc3539b4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.486672 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.486600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:49.552096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.552073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/95aff6cc-a715-473c-8e92-9303dc3539b4-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kfw9\" (UID: \"95aff6cc-a715-473c-8e92-9303dc3539b4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.552244 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.552151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8xfv\" (UniqueName: \"kubernetes.io/projected/95aff6cc-a715-473c-8e92-9303dc3539b4-kube-api-access-b8xfv\") pod \"monitoring-plugin-5d85778b9d-8kfw9\" (UID: \"95aff6cc-a715-473c-8e92-9303dc3539b4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.554745 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.554715 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/95aff6cc-a715-473c-8e92-9303dc3539b4-monitoring-plugin-cert\") pod \"monitoring-plugin-5d85778b9d-8kfw9\" (UID: \"95aff6cc-a715-473c-8e92-9303dc3539b4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.560808 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.560788 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8xfv\" (UniqueName: \"kubernetes.io/projected/95aff6cc-a715-473c-8e92-9303dc3539b4-kube-api-access-b8xfv\") pod \"monitoring-plugin-5d85778b9d-8kfw9\" (UID: \"95aff6cc-a715-473c-8e92-9303dc3539b4\") " pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.572581 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.572562 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:49.805976 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.805949 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-65956475c7-xqv99"] Sep 11 13:54:49.808522 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.808508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.812056 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.812034 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 11 13:54:49.812624 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.812591 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 11 13:54:49.813291 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.813267 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7psx2\"" Sep 11 13:54:49.813365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.813269 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 11 13:54:49.813403 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.813270 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 11 13:54:49.813403 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.813275 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 11 13:54:49.818854 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.818834 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 11 13:54:49.825574 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.825554 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65956475c7-xqv99"] Sep 11 13:54:49.858577 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.858551 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9"] Sep 11 13:54:49.863686 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:49.863651 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95aff6cc_a715_473c_8e92_9303dc3539b4.slice/crio-b6c1e587d2601115408c90d1209e6f7e057e8f32c056d4e4d633406d4469688e WatchSource:0}: Error finding container b6c1e587d2601115408c90d1209e6f7e057e8f32c056d4e4d633406d4469688e: Status 404 returned error can't find the container with id b6c1e587d2601115408c90d1209e6f7e057e8f32c056d4e4d633406d4469688e Sep 11 13:54:49.958239 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-metrics-client-ca\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-secret-telemeter-client\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958283 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-serving-certs-ca-bundle\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958297 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-federate-client-tls\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958359 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-telemeter-client-tls\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958567 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958405 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk727\" (UniqueName: \"kubernetes.io/projected/e1746003-2067-4c36-9dcf-f9bcb4eca44a-kube-api-access-jk727\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958567 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:49.958567 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:49.958497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059478 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-telemeter-client-tls\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059478 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059433 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk727\" (UniqueName: \"kubernetes.io/projected/e1746003-2067-4c36-9dcf-f9bcb4eca44a-kube-api-access-jk727\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059478 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059460 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059734 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059734 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-metrics-client-ca\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059734 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059551 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-secret-telemeter-client\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059734 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059696 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-serving-certs-ca-bundle\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.059734 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.059725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-federate-client-tls\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.060589 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.060545 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-serving-certs-ca-bundle\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.060715 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.060645 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-metrics-client-ca\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.061064 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.061042 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1746003-2067-4c36-9dcf-f9bcb4eca44a-telemeter-trusted-ca-bundle\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.062759 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.062739 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-telemeter-client-tls\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.062835 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.062814 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.063051 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.063036 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-federate-client-tls\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.063093 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.063040 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1746003-2067-4c36-9dcf-f9bcb4eca44a-secret-telemeter-client\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.070416 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.070391 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk727\" (UniqueName: \"kubernetes.io/projected/e1746003-2067-4c36-9dcf-f9bcb4eca44a-kube-api-access-jk727\") pod \"telemeter-client-65956475c7-xqv99\" (UID: \"e1746003-2067-4c36-9dcf-f9bcb4eca44a\") " pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.118828 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.118805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" Sep 11 13:54:50.205644 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.205618 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" event={"ID":"95aff6cc-a715-473c-8e92-9303dc3539b4","Type":"ContainerStarted","Data":"b6c1e587d2601115408c90d1209e6f7e057e8f32c056d4e4d633406d4469688e"} Sep 11 13:54:50.404116 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:50.404072 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-65956475c7-xqv99"] Sep 11 13:54:50.405642 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:50.405614 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1746003_2067_4c36_9dcf_f9bcb4eca44a.slice/crio-337f7d17b328ba59499e7256d631e85b11c2c662267ebe083acdd376d8f30b9d WatchSource:0}: Error finding container 337f7d17b328ba59499e7256d631e85b11c2c662267ebe083acdd376d8f30b9d: Status 404 returned error can't find the container with id 337f7d17b328ba59499e7256d631e85b11c2c662267ebe083acdd376d8f30b9d Sep 11 13:54:51.213572 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:51.213149 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" event={"ID":"95aff6cc-a715-473c-8e92-9303dc3539b4","Type":"ContainerStarted","Data":"a09e3802e809ba797c43dec17eda4a2f66f6585b408b3356e1032ced716b4f85"} Sep 11 13:54:51.213572 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:51.213217 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:51.217334 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:51.217295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" event={"ID":"e1746003-2067-4c36-9dcf-f9bcb4eca44a","Type":"ContainerStarted","Data":"337f7d17b328ba59499e7256d631e85b11c2c662267ebe083acdd376d8f30b9d"} Sep 11 13:54:51.218039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:51.218005 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:51.223292 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:51.223248 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" Sep 11 13:54:51.241283 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:51.241228 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-5d85778b9d-8kfw9" podStartSLOduration=0.991361616 podStartE2EDuration="2.241206223s" totalImagesPullingTime="1.249844607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:51.238436496 +0000 UTC m=+108.108167760" watchObservedRunningTime="2026-09-11 13:54:51.241206223 +0000 UTC m=+108.110937488" Sep 11 13:54:52.746614 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:52.746583 2579 patch_prober.go:28] interesting pod/image-registry-f54f4c8f9-bk2d7 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 13:54:52.746956 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:52.746631 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 13:54:53.159016 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:53.158985 2579 patch_prober.go:28] interesting pod/image-registry-797f7fc5b4-852mw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 11 13:54:53.159180 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:53.159037 2579 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" podUID="d18dcec8-7619-4a8a-b790-15564fb955dc" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 13:54:53.226956 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:53.226919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" event={"ID":"e1746003-2067-4c36-9dcf-f9bcb4eca44a","Type":"ContainerStarted","Data":"5279be39f7bf49d1875fbd877db12a34581f9804c5a6d0567e4b952bd8a05c8d"} Sep 11 13:54:54.179795 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:54.179762 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-g8b9r" Sep 11 13:54:54.232958 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:54.232927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" event={"ID":"e1746003-2067-4c36-9dcf-f9bcb4eca44a","Type":"ContainerStarted","Data":"030e95109806a5d9dc3a17bab78faeafafa45070f17d868d7728dc868ce76469"} Sep 11 13:54:54.233138 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:54.232987 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" event={"ID":"e1746003-2067-4c36-9dcf-f9bcb4eca44a","Type":"ContainerStarted","Data":"9855253fb383cc2676bc5d885ee2f75f519d4779a967df55abb4a5d47ad28fc6"} Sep 11 13:54:54.255884 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:54.255835 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-65956475c7-xqv99" podStartSLOduration=2.209713799 podStartE2EDuration="5.25581643s" totalImagesPullingTime="3.046102631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:54.255245468 +0000 UTC m=+111.124976730" watchObservedRunningTime="2026-09-11 13:54:54.25581643 +0000 UTC m=+111.125547712" Sep 11 13:54:55.139844 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:55.139817 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-797f7fc5b4-852mw" Sep 11 13:54:55.356767 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:55.356737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:55.440012 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:55.439921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:55.540076 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:55.540044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:57.763232 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:57.763197 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" containerID="cri-o://373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c" gracePeriod=30 Sep 11 13:54:57.843983 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:57.843955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:57.942969 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:57.942847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:57.999936 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:57.999915 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:58.019324 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019272 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019324 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019305 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-trusted-ca\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019446 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019327 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-image-registry-private-configuration\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019446 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019353 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-ca-trust-extracted\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019446 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019399 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-bound-sa-token\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019465 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5b6k\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-kube-api-access-c5b6k\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019497 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-certificates\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019526 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-installation-pull-secrets\") pod \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\" (UID: \"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7\") " Sep 11 13:54:58.019868 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019765 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:54:58.020053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.019994 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:54:58.021912 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.021869 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:54:58.021998 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.021912 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:54:58.021998 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.021919 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:54:58.022521 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.022493 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-kube-api-access-c5b6k" (OuterVolumeSpecName: "kube-api-access-c5b6k") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "kube-api-access-c5b6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:54:58.022521 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.022503 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:54:58.027560 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.027542 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-76f99584f4-4pmwh"] Sep 11 13:54:58.027866 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.027852 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" Sep 11 13:54:58.027939 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.027870 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" Sep 11 13:54:58.027993 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.027986 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerName="registry" Sep 11 13:54:58.028885 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.028860 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" (UID: "ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:54:58.032384 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.032368 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:54:58.035028 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.035011 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-2ngcq\"" Sep 11 13:54:58.035365 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.035351 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 11 13:54:58.035844 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.035829 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 11 13:54:58.046859 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.046839 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-4pmwh"] Sep 11 13:54:58.047243 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.047220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:58.120971 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.120948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2xsc\" (UniqueName: \"kubernetes.io/projected/ae9357c0-5a13-4f62-85f8-8cda41eb985c-kube-api-access-z2xsc\") pod \"downloads-76f99584f4-4pmwh\" (UID: \"ae9357c0-5a13-4f62-85f8-8cda41eb985c\") " pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:54:58.121061 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121019 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-trusted-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121061 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121034 2579 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-image-registry-private-configuration\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121061 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121043 2579 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-ca-trust-extracted\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121061 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121051 2579 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-bound-sa-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121061 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121060 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5b6k\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-kube-api-access-c5b6k\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121249 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121068 2579 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-certificates\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121249 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121076 2579 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-installation-pull-secrets\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.121249 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.121084 2579 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7-registry-tls\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:54:58.149978 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.149943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:58.222050 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.222023 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2xsc\" (UniqueName: \"kubernetes.io/projected/ae9357c0-5a13-4f62-85f8-8cda41eb985c-kube-api-access-z2xsc\") pod \"downloads-76f99584f4-4pmwh\" (UID: \"ae9357c0-5a13-4f62-85f8-8cda41eb985c\") " pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:54:58.230738 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.230710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2xsc\" (UniqueName: \"kubernetes.io/projected/ae9357c0-5a13-4f62-85f8-8cda41eb985c-kube-api-access-z2xsc\") pod \"downloads-76f99584f4-4pmwh\" (UID: \"ae9357c0-5a13-4f62-85f8-8cda41eb985c\") " pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:54:58.248620 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.248582 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" containerID="373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c" exitCode=0 Sep 11 13:54:58.248719 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.248630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" Sep 11 13:54:58.248719 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.248641 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" event={"ID":"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7","Type":"ContainerDied","Data":"373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c"} Sep 11 13:54:58.248719 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.248684 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-f54f4c8f9-bk2d7" event={"ID":"ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7","Type":"ContainerDied","Data":"910e2527eea671720294ed022c9ffba0f6478c43d8b19f0884742cc7b7c81a17"} Sep 11 13:54:58.248719 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.248708 2579 scope.go:117] "RemoveContainer" containerID="373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c" Sep 11 13:54:58.257647 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.257632 2579 scope.go:117] "RemoveContainer" containerID="373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c" Sep 11 13:54:58.257910 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:54:58.257889 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c\": container with ID starting with 373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c not found: ID does not exist" containerID="373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c" Sep 11 13:54:58.257969 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.257918 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c"} err="failed to get container status \"373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c\": rpc error: code = NotFound desc = could not find container \"373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c\": container with ID starting with 373ec6f230be043f3daf84af7715a3c69ebefb04fb46e614e07288a12cee883c not found: ID does not exist" Sep 11 13:54:58.271883 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.271830 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-f54f4c8f9-bk2d7"] Sep 11 13:54:58.274681 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.274662 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-f54f4c8f9-bk2d7"] Sep 11 13:54:58.320735 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.320705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:58.341774 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.341753 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:54:58.498891 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.498856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:58.574617 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.574561 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:58.614123 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.614078 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-76f99584f4-4pmwh"] Sep 11 13:54:58.617805 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:54:58.617781 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae9357c0_5a13_4f62_85f8_8cda41eb985c.slice/crio-db0f88625c0cc250071cb050c56f6108067524b8070825d82d7c9a2fab66d50e WatchSource:0}: Error finding container db0f88625c0cc250071cb050c56f6108067524b8070825d82d7c9a2fab66d50e: Status 404 returned error can't find the container with id db0f88625c0cc250071cb050c56f6108067524b8070825d82d7c9a2fab66d50e Sep 11 13:54:58.944291 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:58.944198 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:59.044382 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.044347 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:59.151385 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.151341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:59.235641 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.235551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:59.257791 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.257753 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-4pmwh" event={"ID":"ae9357c0-5a13-4f62-85f8-8cda41eb985c","Type":"ContainerStarted","Data":"db0f88625c0cc250071cb050c56f6108067524b8070825d82d7c9a2fab66d50e"} Sep 11 13:54:59.640830 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.640799 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7" path="/var/lib/kubelet/pods/ba4a84c6-9cfe-41d0-bbfc-2d8236d687d7/volumes" Sep 11 13:54:59.724151 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.724097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:54:59.923422 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:54:59.923341 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:00.090921 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:00.090891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:00.160927 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:00.160893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:01.046615 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:01.046573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:01.148750 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:01.148721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:01.231152 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:01.231067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:02.997558 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:02.997521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:03.060755 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:03.060714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:04.184769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:04.184734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:04.257614 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:04.257579 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:06.855818 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:06.855787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:06.958920 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:06.958867 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:08.174398 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:08.174361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:08.407677 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:08.407642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:09.056355 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.056322 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-585b78867f-2t9k8"] Sep 11 13:55:09.061584 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.061557 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.079682 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.079654 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 11 13:55:09.079813 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.079697 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 11 13:55:09.080429 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.080414 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 11 13:55:09.085747 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.085734 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 11 13:55:09.085823 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.085808 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-kmtnw\"" Sep 11 13:55:09.088957 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.088942 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 11 13:55:09.102587 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.102567 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585b78867f-2t9k8"] Sep 11 13:55:09.118346 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.118326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-serving-cert\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.118476 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.118357 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-service-ca\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.118476 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.118391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbjmt\" (UniqueName: \"kubernetes.io/projected/721d85b3-d073-40e0-bfc3-46593a8a4073-kube-api-access-wbjmt\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.118476 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.118461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-console-config\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.118592 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.118513 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-oauth-serving-cert\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.118592 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.118539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-oauth-config\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.219465 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.219429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-oauth-serving-cert\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.219889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.219485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-oauth-config\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.219889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.219546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-serving-cert\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.219889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.219563 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-service-ca\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.219889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.219592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbjmt\" (UniqueName: \"kubernetes.io/projected/721d85b3-d073-40e0-bfc3-46593a8a4073-kube-api-access-wbjmt\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.219889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.219638 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-console-config\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.220179 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.220126 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-oauth-serving-cert\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.220316 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.220282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-console-config\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.220729 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.220704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-service-ca\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.222777 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.222760 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-oauth-config\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.223016 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.223000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-serving-cert\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.238088 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.238056 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbjmt\" (UniqueName: \"kubernetes.io/projected/721d85b3-d073-40e0-bfc3-46593a8a4073-kube-api-access-wbjmt\") pod \"console-585b78867f-2t9k8\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.372260 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.372170 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:09.656437 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:09.656354 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-585b78867f-2t9k8"] Sep 11 13:55:09.659660 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:55:09.659630 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721d85b3_d073_40e0_bfc3_46593a8a4073.slice/crio-b7af846b9f5616fc290c38a4df1d2503143a37b5ab9fd327e2e796d301cf5e4c WatchSource:0}: Error finding container b7af846b9f5616fc290c38a4df1d2503143a37b5ab9fd327e2e796d301cf5e4c: Status 404 returned error can't find the container with id b7af846b9f5616fc290c38a4df1d2503143a37b5ab9fd327e2e796d301cf5e4c Sep 11 13:55:10.307666 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:10.307632 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585b78867f-2t9k8" event={"ID":"721d85b3-d073-40e0-bfc3-46593a8a4073","Type":"ContainerStarted","Data":"b7af846b9f5616fc290c38a4df1d2503143a37b5ab9fd327e2e796d301cf5e4c"} Sep 11 13:55:11.228150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:11.228091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:11.356992 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:11.356955 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:11.545263 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:11.545207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:12.437828 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:12.437797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:12.527042 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:12.527008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:13.456308 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:13.456275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:55:13.459020 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:13.458994 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8d081982-c02f-4f76-a910-bd0406b66e19-metrics-certs\") pod \"network-metrics-daemon-zbh64\" (UID: \"8d081982-c02f-4f76-a910-bd0406b66e19\") " pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:55:13.735611 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:13.735523 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:13.751914 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:13.751889 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-k94vs\"" Sep 11 13:55:13.760229 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:13.760208 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-zbh64" Sep 11 13:55:13.896513 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:13.896472 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:16.050711 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:16.050670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:16.151827 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:16.151794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:16.969572 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:16.969505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-zbh64"] Sep 11 13:55:16.972979 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:55:16.972938 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d081982_c02f_4f76_a910_bd0406b66e19.slice/crio-3595fd89a9cdda51048f7ecb4f1685b8cda7199cee69879a0452cb96a6d844c2 WatchSource:0}: Error finding container 3595fd89a9cdda51048f7ecb4f1685b8cda7199cee69879a0452cb96a6d844c2: Status 404 returned error can't find the container with id 3595fd89a9cdda51048f7ecb4f1685b8cda7199cee69879a0452cb96a6d844c2 Sep 11 13:55:17.175753 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.175717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:17.294997 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.294944 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:17.342483 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.342445 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-76f99584f4-4pmwh" event={"ID":"ae9357c0-5a13-4f62-85f8-8cda41eb985c","Type":"ContainerStarted","Data":"cc9e13db0c09ee701108130cf26e89c70ec888b63a77d80f0ad922e009c1c31a"} Sep 11 13:55:17.343505 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.343438 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:55:17.346697 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.346664 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbh64" event={"ID":"8d081982-c02f-4f76-a910-bd0406b66e19","Type":"ContainerStarted","Data":"3595fd89a9cdda51048f7ecb4f1685b8cda7199cee69879a0452cb96a6d844c2"} Sep 11 13:55:17.348387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.348362 2579 generic.go:356] "Generic (PLEG): container finished" podID="9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb" containerID="8d32b49ca792085ff481f2a522efb5eeaf2775af9a4c650c7e0f2a84d6be288a" exitCode=0 Sep 11 13:55:17.348487 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.348401 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" event={"ID":"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb","Type":"ContainerDied","Data":"8d32b49ca792085ff481f2a522efb5eeaf2775af9a4c650c7e0f2a84d6be288a"} Sep 11 13:55:17.348974 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.348957 2579 scope.go:117] "RemoveContainer" containerID="8d32b49ca792085ff481f2a522efb5eeaf2775af9a4c650c7e0f2a84d6be288a" Sep 11 13:55:17.356398 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.356376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-76f99584f4-4pmwh" Sep 11 13:55:17.362737 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.362712 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77c6bd5cf8-qlf5s"] Sep 11 13:55:17.367830 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.367474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.380513 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.380489 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 11 13:55:17.396902 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.396845 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c6bd5cf8-qlf5s"] Sep 11 13:55:17.439598 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.439559 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-76f99584f4-4pmwh" podStartSLOduration=1.26603291 podStartE2EDuration="19.439537432s" totalImagesPullingTime="18.173504522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:17.407934114 +0000 UTC m=+134.277665378" watchObservedRunningTime="2026-09-11 13:55:17.439537432 +0000 UTC m=+134.309268693" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvlj\" (UniqueName: \"kubernetes.io/projected/649d1cbc-0289-4893-8f81-211689508b42-kube-api-access-vtvlj\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497328 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-console-config\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-serving-cert\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497421 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-oauth-serving-cert\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-service-ca\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-oauth-config\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.499602 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.497595 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-trusted-ca-bundle\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598393 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-oauth-config\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598457 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-trusted-ca-bundle\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvlj\" (UniqueName: \"kubernetes.io/projected/649d1cbc-0289-4893-8f81-211689508b42-kube-api-access-vtvlj\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598532 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-console-config\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-serving-cert\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598590 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-oauth-serving-cert\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599164 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.598627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-service-ca\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.599639 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.599431 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-service-ca\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.600404 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.600376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-console-config\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.602545 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.601365 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-trusted-ca-bundle\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.602545 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.602504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-oauth-serving-cert\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.604444 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.604416 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-oauth-config\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.606900 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.606862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-serving-cert\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.611358 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.611316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvlj\" (UniqueName: \"kubernetes.io/projected/649d1cbc-0289-4893-8f81-211689508b42-kube-api-access-vtvlj\") pod \"console-77c6bd5cf8-qlf5s\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:17.685891 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:17.685852 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:18.013117 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:18.012208 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77c6bd5cf8-qlf5s"] Sep 11 13:55:18.331333 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:55:18.331242 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649d1cbc_0289_4893_8f81_211689508b42.slice/crio-ef7b6b99cf4491427aa748d283d800092d36399aa5e7ad13c92a892514daac48 WatchSource:0}: Error finding container ef7b6b99cf4491427aa748d283d800092d36399aa5e7ad13c92a892514daac48: Status 404 returned error can't find the container with id ef7b6b99cf4491427aa748d283d800092d36399aa5e7ad13c92a892514daac48 Sep 11 13:55:18.358314 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:18.355657 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-cf7c776b7-sh4lj" event={"ID":"9c0ddc27-1cbd-40ce-afcf-1a6967dedfdb","Type":"ContainerStarted","Data":"8bee95f889d7d579667fd454995f5b436e5bf0c97a59cc27f260a39b3807b2db"} Sep 11 13:55:18.358314 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:18.357356 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c6bd5cf8-qlf5s" event={"ID":"649d1cbc-0289-4893-8f81-211689508b42","Type":"ContainerStarted","Data":"ef7b6b99cf4491427aa748d283d800092d36399aa5e7ad13c92a892514daac48"} Sep 11 13:55:18.765345 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:18.765312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:18.868658 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:18.868549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:19.367234 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:19.367188 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbh64" event={"ID":"8d081982-c02f-4f76-a910-bd0406b66e19","Type":"ContainerStarted","Data":"12633319dd0b9a84deac842e35a1bf940c47599346a60091de0f3dbb3c4a2a5c"} Sep 11 13:55:19.368056 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:19.368002 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-zbh64" event={"ID":"8d081982-c02f-4f76-a910-bd0406b66e19","Type":"ContainerStarted","Data":"b249f9a405e8049de4ec888875126cc5236e21f91ffbee50b9d213cb76476705"} Sep 11 13:55:19.384982 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:19.384931 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-zbh64" podStartSLOduration=133.932013415 podStartE2EDuration="2m15.384913779s" totalImagesPullingTime="1.452900364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:19.383643276 +0000 UTC m=+136.253374541" watchObservedRunningTime="2026-09-11 13:55:19.384913779 +0000 UTC m=+136.254645042" Sep 11 13:55:19.514613 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:19.514418 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:19.594232 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:19.594200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:22.386769 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:22.386727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585b78867f-2t9k8" event={"ID":"721d85b3-d073-40e0-bfc3-46593a8a4073","Type":"ContainerStarted","Data":"a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b"} Sep 11 13:55:22.388569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:22.388537 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c6bd5cf8-qlf5s" event={"ID":"649d1cbc-0289-4893-8f81-211689508b42","Type":"ContainerStarted","Data":"6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a"} Sep 11 13:55:22.409756 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:22.409701 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-585b78867f-2t9k8" podStartSLOduration=1.721367944 podStartE2EDuration="13.409682004s" totalImagesPullingTime="11.68831406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:22.407591491 +0000 UTC m=+139.277322752" watchObservedRunningTime="2026-09-11 13:55:22.409682004 +0000 UTC m=+139.279413265" Sep 11 13:55:22.430547 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:22.429536 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77c6bd5cf8-qlf5s" podStartSLOduration=2.406326512 podStartE2EDuration="5.429518264s" totalImagesPullingTime="3.023191752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:55:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:55:22.42801901 +0000 UTC m=+139.297750272" watchObservedRunningTime="2026-09-11 13:55:22.429518264 +0000 UTC m=+139.299249526" Sep 11 13:55:26.013143 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:26.013085 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:27.415726 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:27.415693 2579 generic.go:356] "Generic (PLEG): container finished" podID="c49eeabe-0790-4d42-9541-ade2388aacde" containerID="de78fb0d0262c1cdd05ced3d7ededb7a93a0e3a61b4f88846aa867cc638a759f" exitCode=0 Sep 11 13:55:27.416086 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:27.415766 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" event={"ID":"c49eeabe-0790-4d42-9541-ade2388aacde","Type":"ContainerDied","Data":"de78fb0d0262c1cdd05ced3d7ededb7a93a0e3a61b4f88846aa867cc638a759f"} Sep 11 13:55:27.416086 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:27.416052 2579 scope.go:117] "RemoveContainer" containerID="de78fb0d0262c1cdd05ced3d7ededb7a93a0e3a61b4f88846aa867cc638a759f" Sep 11 13:55:27.686585 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:27.686503 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:27.686585 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:27.686556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:27.692674 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:27.692646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:28.421685 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:28.421644 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-7k7gv" event={"ID":"c49eeabe-0790-4d42-9541-ade2388aacde","Type":"ContainerStarted","Data":"1bbbb870503136c2badd5409569361a21f8d494ad176e682047dc22d71a94c81"} Sep 11 13:55:28.426069 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:28.426047 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:55:28.493038 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:28.493006 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585b78867f-2t9k8"] Sep 11 13:55:28.610496 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:28.610459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:28.718910 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:28.718821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:28.809298 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:28.809270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:29.230407 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:29.230369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:29.365626 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:29.365597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:29.372302 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:29.372284 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:31.053962 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:31.053926 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:31.155797 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:31.155764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:35.995530 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:35.995490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:36.104570 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:36.104539 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:46.080242 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:46.080208 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:46.181316 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:46.181280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:55:53.512702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.512632 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-585b78867f-2t9k8" podUID="721d85b3-d073-40e0-bfc3-46593a8a4073" containerName="console" containerID="cri-o://a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b" gracePeriod=15 Sep 11 13:55:53.790459 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.790402 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:53.888818 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.888789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-oauth-serving-cert\") pod \"721d85b3-d073-40e0-bfc3-46593a8a4073\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " Sep 11 13:55:53.888942 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.888834 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-serving-cert\") pod \"721d85b3-d073-40e0-bfc3-46593a8a4073\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " Sep 11 13:55:53.888942 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.888855 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-console-config\") pod \"721d85b3-d073-40e0-bfc3-46593a8a4073\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " Sep 11 13:55:53.888942 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.888901 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-service-ca\") pod \"721d85b3-d073-40e0-bfc3-46593a8a4073\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " Sep 11 13:55:53.888942 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.888916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wbjmt\" (UniqueName: \"kubernetes.io/projected/721d85b3-d073-40e0-bfc3-46593a8a4073-kube-api-access-wbjmt\") pod \"721d85b3-d073-40e0-bfc3-46593a8a4073\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " Sep 11 13:55:53.889148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.888948 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-oauth-config\") pod \"721d85b3-d073-40e0-bfc3-46593a8a4073\" (UID: \"721d85b3-d073-40e0-bfc3-46593a8a4073\") " Sep 11 13:55:53.889281 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.889250 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "721d85b3-d073-40e0-bfc3-46593a8a4073" (UID: "721d85b3-d073-40e0-bfc3-46593a8a4073"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:53.889388 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.889301 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-service-ca" (OuterVolumeSpecName: "service-ca") pod "721d85b3-d073-40e0-bfc3-46593a8a4073" (UID: "721d85b3-d073-40e0-bfc3-46593a8a4073"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:53.889388 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.889310 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-console-config" (OuterVolumeSpecName: "console-config") pod "721d85b3-d073-40e0-bfc3-46593a8a4073" (UID: "721d85b3-d073-40e0-bfc3-46593a8a4073"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:55:53.891278 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.891251 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "721d85b3-d073-40e0-bfc3-46593a8a4073" (UID: "721d85b3-d073-40e0-bfc3-46593a8a4073"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:55:53.891278 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.891263 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "721d85b3-d073-40e0-bfc3-46593a8a4073" (UID: "721d85b3-d073-40e0-bfc3-46593a8a4073"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:55:53.891415 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.891392 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/721d85b3-d073-40e0-bfc3-46593a8a4073-kube-api-access-wbjmt" (OuterVolumeSpecName: "kube-api-access-wbjmt") pod "721d85b3-d073-40e0-bfc3-46593a8a4073" (UID: "721d85b3-d073-40e0-bfc3-46593a8a4073"). InnerVolumeSpecName "kube-api-access-wbjmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:55:53.989714 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.989688 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-oauth-config\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:55:53.989714 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.989711 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-oauth-serving-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:55:53.989862 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.989724 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/721d85b3-d073-40e0-bfc3-46593a8a4073-console-serving-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:55:53.989862 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.989736 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-console-config\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:55:53.989862 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.989748 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/721d85b3-d073-40e0-bfc3-46593a8a4073-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:55:53.989862 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:53.989759 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbjmt\" (UniqueName: \"kubernetes.io/projected/721d85b3-d073-40e0-bfc3-46593a8a4073-kube-api-access-wbjmt\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:55:54.523467 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.523432 2579 generic.go:356] "Generic (PLEG): container finished" podID="721d85b3-d073-40e0-bfc3-46593a8a4073" containerID="a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b" exitCode=0 Sep 11 13:55:54.523873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.523473 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585b78867f-2t9k8" event={"ID":"721d85b3-d073-40e0-bfc3-46593a8a4073","Type":"ContainerDied","Data":"a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b"} Sep 11 13:55:54.523873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.523496 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-585b78867f-2t9k8" Sep 11 13:55:54.523873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.523532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-585b78867f-2t9k8" event={"ID":"721d85b3-d073-40e0-bfc3-46593a8a4073","Type":"ContainerDied","Data":"b7af846b9f5616fc290c38a4df1d2503143a37b5ab9fd327e2e796d301cf5e4c"} Sep 11 13:55:54.523873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.523560 2579 scope.go:117] "RemoveContainer" containerID="a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b" Sep 11 13:55:54.532517 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.532500 2579 scope.go:117] "RemoveContainer" containerID="a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b" Sep 11 13:55:54.532753 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:55:54.532738 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b\": container with ID starting with a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b not found: ID does not exist" containerID="a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b" Sep 11 13:55:54.532804 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.532759 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b"} err="failed to get container status \"a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b\": rpc error: code = NotFound desc = could not find container \"a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b\": container with ID starting with a760d8c98c011ba779f91b500a9b0af3005d1f638d8c171811f256066366d25b not found: ID does not exist" Sep 11 13:55:54.553327 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.553309 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-585b78867f-2t9k8"] Sep 11 13:55:54.562411 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:54.562393 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-585b78867f-2t9k8"] Sep 11 13:55:55.640772 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:55:55.640744 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="721d85b3-d073-40e0-bfc3-46593a8a4073" path="/var/lib/kubelet/pods/721d85b3-d073-40e0-bfc3-46593a8a4073/volumes" Sep 11 13:56:01.102189 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:01.102157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:01.158309 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:01.158248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:07.171252 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.171217 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 13:56:07.171681 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.171527 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="721d85b3-d073-40e0-bfc3-46593a8a4073" containerName="console" Sep 11 13:56:07.171681 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.171539 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="721d85b3-d073-40e0-bfc3-46593a8a4073" containerName="console" Sep 11 13:56:07.171681 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.171598 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="721d85b3-d073-40e0-bfc3-46593a8a4073" containerName="console" Sep 11 13:56:07.174928 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.174899 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.177386 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.177359 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-vsv7n\"" Sep 11 13:56:07.177751 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.177547 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 11 13:56:07.177751 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.177622 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 11 13:56:07.177909 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.177841 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 11 13:56:07.177909 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.177890 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 11 13:56:07.178330 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.178027 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 11 13:56:07.178330 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.178214 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 11 13:56:07.178496 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.178368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 11 13:56:07.178496 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.178477 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 11 13:56:07.183505 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.183481 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 11 13:56:07.188973 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.188953 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 13:56:07.267448 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.267414 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7b9dc8fd87-vhjcc"] Sep 11 13:56:07.269539 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.269524 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.284258 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.284228 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b9dc8fd87-vhjcc"] Sep 11 13:56:07.288763 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.288732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf7a111e-28f9-4f43-9089-a93bbd485e25-config-out\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.288889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.288776 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-web-config\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.288889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.288805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.288889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.288829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289027 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.288935 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-config-volume\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289027 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.288971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7a111e-28f9-4f43-9089-a93bbd485e25-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289027 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289001 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf7a111e-28f9-4f43-9089-a93bbd485e25-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289228 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289079 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289228 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289144 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289228 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289195 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a111e-28f9-4f43-9089-a93bbd485e25-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289346 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289227 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cf7a111e-28f9-4f43-9089-a93bbd485e25-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289346 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7f62\" (UniqueName: \"kubernetes.io/projected/cf7a111e-28f9-4f43-9089-a93bbd485e25-kube-api-access-p7f62\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.289346 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.289308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.314493 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.314464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:07.381124 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.381074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:07.390496 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfzgt\" (UniqueName: \"kubernetes.io/projected/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-kube-api-access-xfzgt\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.390588 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390503 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-service-ca\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.390588 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-config-volume\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.390588 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7a111e-28f9-4f43-9089-a93bbd485e25-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.390588 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390580 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf7a111e-28f9-4f43-9089-a93bbd485e25-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.390800 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-oauth-serving-cert\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.390800 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.390800 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.390800 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a111e-28f9-4f43-9089-a93bbd485e25-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.390800 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-serving-cert\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.390800 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-config\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390813 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cf7a111e-28f9-4f43-9089-a93bbd485e25-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7f62\" (UniqueName: \"kubernetes.io/projected/cf7a111e-28f9-4f43-9089-a93bbd485e25-kube-api-access-p7f62\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-trusted-ca-bundle\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf7a111e-28f9-4f43-9089-a93bbd485e25-config-out\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.390973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-oauth-config\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.391010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-web-config\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.391039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391128 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.391064 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391543 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.391201 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cf7a111e-28f9-4f43-9089-a93bbd485e25-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.391604 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.391560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cf7a111e-28f9-4f43-9089-a93bbd485e25-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.392302 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.392279 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cf7a111e-28f9-4f43-9089-a93bbd485e25-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.394004 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.393952 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-config-volume\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.394469 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.394367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.394854 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.394568 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cf7a111e-28f9-4f43-9089-a93bbd485e25-config-out\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.394854 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.394590 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cf7a111e-28f9-4f43-9089-a93bbd485e25-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.394854 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.394723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.395015 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.394887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.395015 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.395008 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-web-config\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.395563 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.395541 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.395960 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.395939 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cf7a111e-28f9-4f43-9089-a93bbd485e25-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.399427 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.399410 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7f62\" (UniqueName: \"kubernetes.io/projected/cf7a111e-28f9-4f43-9089-a93bbd485e25-kube-api-access-p7f62\") pod \"alertmanager-main-0\" (UID: \"cf7a111e-28f9-4f43-9089-a93bbd485e25\") " pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.486894 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.486842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 11 13:56:07.491632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491612 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfzgt\" (UniqueName: \"kubernetes.io/projected/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-kube-api-access-xfzgt\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.491718 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-service-ca\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.491718 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-oauth-serving-cert\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.491819 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491729 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-serving-cert\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.491819 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491751 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-config\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.491942 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491878 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-trusted-ca-bundle\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.491996 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.491955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-oauth-config\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.492453 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.492392 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-service-ca\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.492453 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.492448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-oauth-serving-cert\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.492630 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.492611 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-config\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.492742 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.492722 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-trusted-ca-bundle\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.494166 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.494148 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-serving-cert\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.494404 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.494360 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-oauth-config\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.502839 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.502821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfzgt\" (UniqueName: \"kubernetes.io/projected/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-kube-api-access-xfzgt\") pod \"console-7b9dc8fd87-vhjcc\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.578793 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.578759 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:07.777203 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.777176 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 11 13:56:07.778596 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:56:07.778567 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf7a111e_28f9_4f43_9089_a93bbd485e25.slice/crio-e959c354dab01af716a54c3f729f73b25cdf5ab9a5195636921999b86bffe1cb WatchSource:0}: Error finding container e959c354dab01af716a54c3f729f73b25cdf5ab9a5195636921999b86bffe1cb: Status 404 returned error can't find the container with id e959c354dab01af716a54c3f729f73b25cdf5ab9a5195636921999b86bffe1cb Sep 11 13:56:07.856233 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:07.856207 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7b9dc8fd87-vhjcc"] Sep 11 13:56:07.905892 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:56:07.905868 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc1ccd0_a84a_4930_9bb4_48d2c0a17f07.slice/crio-eb012d57ee88b948ee6bbbe0094a904637670f77ec0c92020d67fdf385fdf0e2 WatchSource:0}: Error finding container eb012d57ee88b948ee6bbbe0094a904637670f77ec0c92020d67fdf385fdf0e2: Status 404 returned error can't find the container with id eb012d57ee88b948ee6bbbe0094a904637670f77ec0c92020d67fdf385fdf0e2 Sep 11 13:56:08.577459 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:08.577426 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9dc8fd87-vhjcc" event={"ID":"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07","Type":"ContainerStarted","Data":"a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e"} Sep 11 13:56:08.577885 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:08.577480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9dc8fd87-vhjcc" event={"ID":"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07","Type":"ContainerStarted","Data":"eb012d57ee88b948ee6bbbe0094a904637670f77ec0c92020d67fdf385fdf0e2"} Sep 11 13:56:08.578961 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:08.578939 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf7a111e-28f9-4f43-9089-a93bbd485e25" containerID="570ffd84bd63ce93ddd4201fb0f832b0b63b59dd7e1fe4dea8b95860bb85ac2c" exitCode=0 Sep 11 13:56:08.579078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:08.578996 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerDied","Data":"570ffd84bd63ce93ddd4201fb0f832b0b63b59dd7e1fe4dea8b95860bb85ac2c"} Sep 11 13:56:08.579078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:08.579038 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"e959c354dab01af716a54c3f729f73b25cdf5ab9a5195636921999b86bffe1cb"} Sep 11 13:56:08.602389 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:08.602349 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7b9dc8fd87-vhjcc" podStartSLOduration=1.6023355700000002 podStartE2EDuration="1.60233557s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:08.600009852 +0000 UTC m=+185.469741127" watchObservedRunningTime="2026-09-11 13:56:08.60233557 +0000 UTC m=+185.472066832" Sep 11 13:56:10.591144 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:10.591086 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"38454827d20b0ca32232a26885e427cc7810fa888d4344e068d88808845d3597"} Sep 11 13:56:10.591518 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:10.591203 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"b4fd626916babd5fb40ca8c305ac9d25561181a34b2777ffa9c40d2bcadad7e5"} Sep 11 13:56:10.591518 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:10.591230 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"929caacb4b6e896fe109007172dfa1e012088526f47b117c34a248544e258dbc"} Sep 11 13:56:10.591518 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:10.591252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"70958c26845b4281beab9c84952c76ca7976c3e7b45ca4b864efe373f0a0055a"} Sep 11 13:56:10.591518 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:10.591276 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"e94285429262e4ecddfb658fcc04c6f6e0cf6c59192963ada724a1b4b253a5b2"} Sep 11 13:56:13.086507 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:13.086473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:13.187632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:13.187600 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:13.605873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:13.605840 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cf7a111e-28f9-4f43-9089-a93bbd485e25","Type":"ContainerStarted","Data":"a31133370c5938683fbdf9529f4387de8ca69569473dbe0b703f8ea6b365437a"} Sep 11 13:56:13.647343 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:13.647287 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.423603314 podStartE2EDuration="6.647266843s" totalImagesPullingTime="4.208598118s" totalInitContainerRuntime="15.065411ms" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:13.645412127 +0000 UTC m=+190.515143388" watchObservedRunningTime="2026-09-11 13:56:13.647266843 +0000 UTC m=+190.516998107" Sep 11 13:56:13.777446 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:13.777411 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:16.084783 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:16.084743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:16.178570 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:16.178541 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:17.579168 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.579140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:17.579595 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.579401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:17.584286 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.584265 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:17.628978 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.628947 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 13:56:17.680391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.680364 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c6bd5cf8-qlf5s"] Sep 11 13:56:17.795965 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.795932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:17.896760 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.896661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:17.958503 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:17.958466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:56:42.705923 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.705856 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77c6bd5cf8-qlf5s" podUID="649d1cbc-0289-4893-8f81-211689508b42" containerName="console" containerID="cri-o://6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a" gracePeriod=15 Sep 11 13:56:42.954588 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.954562 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:56:42.977951 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.977880 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtvlj\" (UniqueName: \"kubernetes.io/projected/649d1cbc-0289-4893-8f81-211689508b42-kube-api-access-vtvlj\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978052 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.977999 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-oauth-config\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978052 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978028 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-console-config\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978188 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978057 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-serving-cert\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978188 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978087 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-trusted-ca-bundle\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978188 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978134 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-oauth-serving-cert\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978188 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978183 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-service-ca\") pod \"649d1cbc-0289-4893-8f81-211689508b42\" (UID: \"649d1cbc-0289-4893-8f81-211689508b42\") " Sep 11 13:56:42.978509 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-console-config" (OuterVolumeSpecName: "console-config") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:56:42.978905 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978524 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:56:42.978905 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978693 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:56:42.978905 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.978823 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-service-ca" (OuterVolumeSpecName: "service-ca") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 13:56:42.980511 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.980485 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649d1cbc-0289-4893-8f81-211689508b42-kube-api-access-vtvlj" (OuterVolumeSpecName: "kube-api-access-vtvlj") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "kube-api-access-vtvlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:56:42.980865 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.980842 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:56:42.980932 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:42.980873 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "649d1cbc-0289-4893-8f81-211689508b42" (UID: "649d1cbc-0289-4893-8f81-211689508b42"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 13:56:43.079342 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079306 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.079342 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079335 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvlj\" (UniqueName: \"kubernetes.io/projected/649d1cbc-0289-4893-8f81-211689508b42-kube-api-access-vtvlj\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.079342 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079345 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-oauth-config\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.079342 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079353 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-console-config\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.079586 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079362 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/649d1cbc-0289-4893-8f81-211689508b42-console-serving-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.079586 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079369 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-trusted-ca-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.079586 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.079380 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/649d1cbc-0289-4893-8f81-211689508b42-oauth-serving-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:56:43.725794 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.725761 2579 generic.go:356] "Generic (PLEG): container finished" podID="649d1cbc-0289-4893-8f81-211689508b42" containerID="6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a" exitCode=0 Sep 11 13:56:43.726264 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.725818 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77c6bd5cf8-qlf5s" Sep 11 13:56:43.726264 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.725850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c6bd5cf8-qlf5s" event={"ID":"649d1cbc-0289-4893-8f81-211689508b42","Type":"ContainerDied","Data":"6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a"} Sep 11 13:56:43.726264 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.725919 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77c6bd5cf8-qlf5s" event={"ID":"649d1cbc-0289-4893-8f81-211689508b42","Type":"ContainerDied","Data":"ef7b6b99cf4491427aa748d283d800092d36399aa5e7ad13c92a892514daac48"} Sep 11 13:56:43.726264 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.725939 2579 scope.go:117] "RemoveContainer" containerID="6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a" Sep 11 13:56:43.734011 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.734000 2579 scope.go:117] "RemoveContainer" containerID="6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a" Sep 11 13:56:43.734311 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:56:43.734291 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a\": container with ID starting with 6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a not found: ID does not exist" containerID="6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a" Sep 11 13:56:43.734371 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.734317 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a"} err="failed to get container status \"6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a\": rpc error: code = NotFound desc = could not find container \"6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a\": container with ID starting with 6332fadc2973389988bf748c9fe950d1dd93748db075281a4a267606f735916a not found: ID does not exist" Sep 11 13:56:43.744202 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.744177 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77c6bd5cf8-qlf5s"] Sep 11 13:56:43.747560 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:43.747539 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77c6bd5cf8-qlf5s"] Sep 11 13:56:45.641219 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:56:45.641186 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="649d1cbc-0289-4893-8f81-211689508b42" path="/var/lib/kubelet/pods/649d1cbc-0289-4893-8f81-211689508b42/volumes" Sep 11 13:57:09.279730 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:09.279691 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:15.803807 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:15.803773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:15.874563 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:15.874530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:16.445985 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.445944 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg"] Sep 11 13:57:16.446323 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.446310 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="649d1cbc-0289-4893-8f81-211689508b42" containerName="console" Sep 11 13:57:16.446371 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.446325 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="649d1cbc-0289-4893-8f81-211689508b42" containerName="console" Sep 11 13:57:16.446405 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.446390 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="649d1cbc-0289-4893-8f81-211689508b42" containerName="console" Sep 11 13:57:16.450902 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.450883 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.453701 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.453681 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 13:57:16.453791 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.453682 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 13:57:16.454553 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.454537 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29475\"" Sep 11 13:57:16.458688 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.458666 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg"] Sep 11 13:57:16.535758 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.535722 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.535930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.535778 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.535930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.535857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpmrd\" (UniqueName: \"kubernetes.io/projected/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-kube-api-access-gpmrd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.636484 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.636448 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.636652 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.636521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.636652 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.636568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpmrd\" (UniqueName: \"kubernetes.io/projected/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-kube-api-access-gpmrd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.636836 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.636818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.636873 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.636855 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.646461 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.646437 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpmrd\" (UniqueName: \"kubernetes.io/projected/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-kube-api-access-gpmrd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:16.761360 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:16.761280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:17.046923 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:17.046886 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg"] Sep 11 13:57:17.049172 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:57:17.049150 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd34be2f6_930e_4477_a7e8_416d7cc4f4e2.slice/crio-c612a3dcd472dc9fb5af686c67d6151421379dda25aef5d5651ca1f2f21b4a2a WatchSource:0}: Error finding container c612a3dcd472dc9fb5af686c67d6151421379dda25aef5d5651ca1f2f21b4a2a: Status 404 returned error can't find the container with id c612a3dcd472dc9fb5af686c67d6151421379dda25aef5d5651ca1f2f21b4a2a Sep 11 13:57:17.859147 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:17.859094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" event={"ID":"d34be2f6-930e-4477-a7e8-416d7cc4f4e2","Type":"ContainerStarted","Data":"c612a3dcd472dc9fb5af686c67d6151421379dda25aef5d5651ca1f2f21b4a2a"} Sep 11 13:57:23.887761 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:23.887723 2579 generic.go:356] "Generic (PLEG): container finished" podID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerID="7003e1e175dbec81ccbf49cb2e8d6ce1c4972398bb9b3d3dfd7ef7623c266007" exitCode=0 Sep 11 13:57:23.888175 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:23.887803 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" event={"ID":"d34be2f6-930e-4477-a7e8-416d7cc4f4e2","Type":"ContainerDied","Data":"7003e1e175dbec81ccbf49cb2e8d6ce1c4972398bb9b3d3dfd7ef7623c266007"} Sep 11 13:57:26.903290 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:26.903252 2579 generic.go:356] "Generic (PLEG): container finished" podID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerID="b4bd143966957b0e3d265b545dbd587763e00d1c45df0079e15c4cd397571269" exitCode=0 Sep 11 13:57:26.903717 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:26.903349 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" event={"ID":"d34be2f6-930e-4477-a7e8-416d7cc4f4e2","Type":"ContainerDied","Data":"b4bd143966957b0e3d265b545dbd587763e00d1c45df0079e15c4cd397571269"} Sep 11 13:57:33.939341 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:33.939304 2579 generic.go:356] "Generic (PLEG): container finished" podID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerID="26b1f00d00b9d638a9fd3e80dd74f19d559ca1854108ea4d216e4e722ae48324" exitCode=0 Sep 11 13:57:33.939707 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:33.939347 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" event={"ID":"d34be2f6-930e-4477-a7e8-416d7cc4f4e2","Type":"ContainerDied","Data":"26b1f00d00b9d638a9fd3e80dd74f19d559ca1854108ea4d216e4e722ae48324"} Sep 11 13:57:35.067218 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.067193 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:35.200833 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.200750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpmrd\" (UniqueName: \"kubernetes.io/projected/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-kube-api-access-gpmrd\") pod \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " Sep 11 13:57:35.200833 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.200801 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-bundle\") pod \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " Sep 11 13:57:35.201006 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.200845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-util\") pod \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\" (UID: \"d34be2f6-930e-4477-a7e8-416d7cc4f4e2\") " Sep 11 13:57:35.201582 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.201491 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-bundle" (OuterVolumeSpecName: "bundle") pod "d34be2f6-930e-4477-a7e8-416d7cc4f4e2" (UID: "d34be2f6-930e-4477-a7e8-416d7cc4f4e2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:57:35.203089 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.203058 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-kube-api-access-gpmrd" (OuterVolumeSpecName: "kube-api-access-gpmrd") pod "d34be2f6-930e-4477-a7e8-416d7cc4f4e2" (UID: "d34be2f6-930e-4477-a7e8-416d7cc4f4e2"). InnerVolumeSpecName "kube-api-access-gpmrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:57:35.205787 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.205767 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-util" (OuterVolumeSpecName: "util") pod "d34be2f6-930e-4477-a7e8-416d7cc4f4e2" (UID: "d34be2f6-930e-4477-a7e8-416d7cc4f4e2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:57:35.302133 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.302079 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpmrd\" (UniqueName: \"kubernetes.io/projected/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-kube-api-access-gpmrd\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:57:35.302133 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.302133 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:57:35.302133 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.302147 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d34be2f6-930e-4477-a7e8-416d7cc4f4e2-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:57:35.949013 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.948933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" event={"ID":"d34be2f6-930e-4477-a7e8-416d7cc4f4e2","Type":"ContainerDied","Data":"c612a3dcd472dc9fb5af686c67d6151421379dda25aef5d5651ca1f2f21b4a2a"} Sep 11 13:57:35.949013 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.948987 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c612a3dcd472dc9fb5af686c67d6151421379dda25aef5d5651ca1f2f21b4a2a" Sep 11 13:57:35.949013 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:35.948944 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540d9qfg" Sep 11 13:57:37.674763 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:37.674718 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:39.050393 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.050335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:39.721585 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721547 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf"] Sep 11 13:57:39.721868 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721856 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="extract" Sep 11 13:57:39.721868 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721869 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="extract" Sep 11 13:57:39.721975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721879 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="util" Sep 11 13:57:39.721975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721884 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="util" Sep 11 13:57:39.721975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721897 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="pull" Sep 11 13:57:39.721975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721903 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="pull" Sep 11 13:57:39.721975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.721950 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d34be2f6-930e-4477-a7e8-416d7cc4f4e2" containerName="extract" Sep 11 13:57:39.764511 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.764472 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf"] Sep 11 13:57:39.764650 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.764574 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:39.767391 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.767368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-96z6x\"" Sep 11 13:57:39.767635 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.767622 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Sep 11 13:57:39.767928 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.767907 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Sep 11 13:57:39.839556 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.839521 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvnsx\" (UniqueName: \"kubernetes.io/projected/2a130f97-d399-4420-abb9-eca39e3b0014-kube-api-access-nvnsx\") pod \"cert-manager-operator-controller-manager-55c444bf86-9mfzf\" (UID: \"2a130f97-d399-4420-abb9-eca39e3b0014\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:39.839718 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.839593 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a130f97-d399-4420-abb9-eca39e3b0014-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9mfzf\" (UID: \"2a130f97-d399-4420-abb9-eca39e3b0014\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:39.940599 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.940560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a130f97-d399-4420-abb9-eca39e3b0014-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9mfzf\" (UID: \"2a130f97-d399-4420-abb9-eca39e3b0014\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:39.940770 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.940616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvnsx\" (UniqueName: \"kubernetes.io/projected/2a130f97-d399-4420-abb9-eca39e3b0014-kube-api-access-nvnsx\") pod \"cert-manager-operator-controller-manager-55c444bf86-9mfzf\" (UID: \"2a130f97-d399-4420-abb9-eca39e3b0014\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:39.940992 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.940974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a130f97-d399-4420-abb9-eca39e3b0014-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9mfzf\" (UID: \"2a130f97-d399-4420-abb9-eca39e3b0014\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:39.952480 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:39.952456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvnsx\" (UniqueName: \"kubernetes.io/projected/2a130f97-d399-4420-abb9-eca39e3b0014-kube-api-access-nvnsx\") pod \"cert-manager-operator-controller-manager-55c444bf86-9mfzf\" (UID: \"2a130f97-d399-4420-abb9-eca39e3b0014\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:40.074057 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:40.074014 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" Sep 11 13:57:40.421311 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:57:40.421275 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a130f97_d399_4420_abb9_eca39e3b0014.slice/crio-6fe1542c8c623a018aab456e7300e5d5710d0b6e436f5ff90ba4915e6cc1cadb WatchSource:0}: Error finding container 6fe1542c8c623a018aab456e7300e5d5710d0b6e436f5ff90ba4915e6cc1cadb: Status 404 returned error can't find the container with id 6fe1542c8c623a018aab456e7300e5d5710d0b6e436f5ff90ba4915e6cc1cadb Sep 11 13:57:40.421530 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:40.421505 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf"] Sep 11 13:57:40.974002 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:40.973966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" event={"ID":"2a130f97-d399-4420-abb9-eca39e3b0014","Type":"ContainerStarted","Data":"6fe1542c8c623a018aab456e7300e5d5710d0b6e436f5ff90ba4915e6cc1cadb"} Sep 11 13:57:41.573315 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:41.573283 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:41.663265 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:41.663238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:46.469962 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:46.469921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:46.534209 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:46.534155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:50.013636 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:50.013550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" event={"ID":"2a130f97-d399-4420-abb9-eca39e3b0014","Type":"ContainerStarted","Data":"1fe9e9fb626699e2f33bbe8bc18e88809dc26015b45e43abe801d9c362af01de"} Sep 11 13:57:50.037511 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:50.037463 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9mfzf" podStartSLOduration=1.9076817940000002 podStartE2EDuration="11.037449031s" totalImagesPullingTime="9.129767237s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:57:50.037068908 +0000 UTC m=+286.906800171" watchObservedRunningTime="2026-09-11 13:57:50.037449031 +0000 UTC m=+286.907180293" Sep 11 13:57:50.561182 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:50.561139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:50.737529 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:50.737493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:51.379788 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.379755 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7xzjc"] Sep 11 13:57:51.381879 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.381861 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.384312 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.384272 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Sep 11 13:57:51.384450 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.384421 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Sep 11 13:57:51.385276 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.385252 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-tmlbf\"" Sep 11 13:57:51.392668 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.392648 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7xzjc"] Sep 11 13:57:51.423714 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.423693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q82h9\" (UniqueName: \"kubernetes.io/projected/f1487f96-5a51-41e8-845f-728e6cce893a-kube-api-access-q82h9\") pod \"cert-manager-webhook-6486ffd997-7xzjc\" (UID: \"f1487f96-5a51-41e8-845f-728e6cce893a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.423815 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.423762 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1487f96-5a51-41e8-845f-728e6cce893a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7xzjc\" (UID: \"f1487f96-5a51-41e8-845f-728e6cce893a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.524355 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.524327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q82h9\" (UniqueName: \"kubernetes.io/projected/f1487f96-5a51-41e8-845f-728e6cce893a-kube-api-access-q82h9\") pod \"cert-manager-webhook-6486ffd997-7xzjc\" (UID: \"f1487f96-5a51-41e8-845f-728e6cce893a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.524471 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.524450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1487f96-5a51-41e8-845f-728e6cce893a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7xzjc\" (UID: \"f1487f96-5a51-41e8-845f-728e6cce893a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.533140 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.533119 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q82h9\" (UniqueName: \"kubernetes.io/projected/f1487f96-5a51-41e8-845f-728e6cce893a-kube-api-access-q82h9\") pod \"cert-manager-webhook-6486ffd997-7xzjc\" (UID: \"f1487f96-5a51-41e8-845f-728e6cce893a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.533341 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.533324 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f1487f96-5a51-41e8-845f-728e6cce893a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-7xzjc\" (UID: \"f1487f96-5a51-41e8-845f-728e6cce893a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.699659 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.699596 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:57:51.937629 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.937598 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt"] Sep 11 13:57:51.940030 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.940009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:51.943242 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.943223 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29475\"" Sep 11 13:57:51.943706 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.943692 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 13:57:51.944355 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.944342 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 13:57:51.950455 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.950402 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt"] Sep 11 13:57:51.968916 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:51.968897 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-7xzjc"] Sep 11 13:57:51.971621 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:57:51.971598 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1487f96_5a51_41e8_845f_728e6cce893a.slice/crio-dcdf8e902d5a33e79d3c0cde7a6695bc0edaa52387e4993c8e5ac879e49aab0a WatchSource:0}: Error finding container dcdf8e902d5a33e79d3c0cde7a6695bc0edaa52387e4993c8e5ac879e49aab0a: Status 404 returned error can't find the container with id dcdf8e902d5a33e79d3c0cde7a6695bc0edaa52387e4993c8e5ac879e49aab0a Sep 11 13:57:52.023185 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.023154 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" event={"ID":"f1487f96-5a51-41e8-845f-728e6cce893a","Type":"ContainerStarted","Data":"dcdf8e902d5a33e79d3c0cde7a6695bc0edaa52387e4993c8e5ac879e49aab0a"} Sep 11 13:57:52.029563 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.029540 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlztl\" (UniqueName: \"kubernetes.io/projected/db1594fe-1404-469a-adb9-0f490b4e4191-kube-api-access-dlztl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.029638 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.029588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.029638 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.029608 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.130512 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.130485 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlztl\" (UniqueName: \"kubernetes.io/projected/db1594fe-1404-469a-adb9-0f490b4e4191-kube-api-access-dlztl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.130628 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.130553 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.130628 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.130588 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.130907 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.130887 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.130974 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.130919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.144114 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.144080 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlztl\" (UniqueName: \"kubernetes.io/projected/db1594fe-1404-469a-adb9-0f490b4e4191-kube-api-access-dlztl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.250224 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.250177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:52.523086 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.523014 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt"] Sep 11 13:57:52.526018 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:57:52.525972 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb1594fe_1404_469a_adb9_0f490b4e4191.slice/crio-c07115666dafe3d728c1df2aa6f88e9f9bbce1c7bb90c17c68d1fe6af563ab6d WatchSource:0}: Error finding container c07115666dafe3d728c1df2aa6f88e9f9bbce1c7bb90c17c68d1fe6af563ab6d: Status 404 returned error can't find the container with id c07115666dafe3d728c1df2aa6f88e9f9bbce1c7bb90c17c68d1fe6af563ab6d Sep 11 13:57:52.851340 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:52.851316 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:53.028791 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:53.028761 2579 generic.go:356] "Generic (PLEG): container finished" podID="db1594fe-1404-469a-adb9-0f490b4e4191" containerID="98a19d5136d6f7a40a2d0fa8ed01ebbf0d09216e2dc29aaa00dedf22be47a908" exitCode=0 Sep 11 13:57:53.028953 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:53.028850 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" event={"ID":"db1594fe-1404-469a-adb9-0f490b4e4191","Type":"ContainerDied","Data":"98a19d5136d6f7a40a2d0fa8ed01ebbf0d09216e2dc29aaa00dedf22be47a908"} Sep 11 13:57:53.028953 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:53.028924 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" event={"ID":"db1594fe-1404-469a-adb9-0f490b4e4191","Type":"ContainerStarted","Data":"c07115666dafe3d728c1df2aa6f88e9f9bbce1c7bb90c17c68d1fe6af563ab6d"} Sep 11 13:57:53.033912 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:53.033889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:57:56.043886 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:56.043849 2579 generic.go:356] "Generic (PLEG): container finished" podID="db1594fe-1404-469a-adb9-0f490b4e4191" containerID="eca79a336f69d26df110b8fbe5fdbc37c63394ea3b35698e9a4e0e24c1852067" exitCode=0 Sep 11 13:57:56.044305 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:56.043936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" event={"ID":"db1594fe-1404-469a-adb9-0f490b4e4191","Type":"ContainerDied","Data":"eca79a336f69d26df110b8fbe5fdbc37c63394ea3b35698e9a4e0e24c1852067"} Sep 11 13:57:57.050912 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:57.050878 2579 generic.go:356] "Generic (PLEG): container finished" podID="db1594fe-1404-469a-adb9-0f490b4e4191" containerID="8ab56e6c4ba5abda2db4455a3b211948bf67a756ad4d6b826823a1d88d8f4563" exitCode=0 Sep 11 13:57:57.051385 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:57.050950 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" event={"ID":"db1594fe-1404-469a-adb9-0f490b4e4191","Type":"ContainerDied","Data":"8ab56e6c4ba5abda2db4455a3b211948bf67a756ad4d6b826823a1d88d8f4563"} Sep 11 13:57:58.178640 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.178616 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:58.279546 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.279510 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlztl\" (UniqueName: \"kubernetes.io/projected/db1594fe-1404-469a-adb9-0f490b4e4191-kube-api-access-dlztl\") pod \"db1594fe-1404-469a-adb9-0f490b4e4191\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " Sep 11 13:57:58.279704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.279578 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-bundle\") pod \"db1594fe-1404-469a-adb9-0f490b4e4191\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " Sep 11 13:57:58.279704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.279684 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-util\") pod \"db1594fe-1404-469a-adb9-0f490b4e4191\" (UID: \"db1594fe-1404-469a-adb9-0f490b4e4191\") " Sep 11 13:57:58.279945 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.279926 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-bundle" (OuterVolumeSpecName: "bundle") pod "db1594fe-1404-469a-adb9-0f490b4e4191" (UID: "db1594fe-1404-469a-adb9-0f490b4e4191"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:57:58.281711 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.281690 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db1594fe-1404-469a-adb9-0f490b4e4191-kube-api-access-dlztl" (OuterVolumeSpecName: "kube-api-access-dlztl") pod "db1594fe-1404-469a-adb9-0f490b4e4191" (UID: "db1594fe-1404-469a-adb9-0f490b4e4191"). InnerVolumeSpecName "kube-api-access-dlztl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:57:58.284079 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.284054 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-util" (OuterVolumeSpecName: "util") pod "db1594fe-1404-469a-adb9-0f490b4e4191" (UID: "db1594fe-1404-469a-adb9-0f490b4e4191"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:57:58.380630 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.380571 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:57:58.380630 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.380600 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlztl\" (UniqueName: \"kubernetes.io/projected/db1594fe-1404-469a-adb9-0f490b4e4191-kube-api-access-dlztl\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:57:58.380630 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:58.380612 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/db1594fe-1404-469a-adb9-0f490b4e4191-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:57:59.062000 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:59.061961 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" event={"ID":"db1594fe-1404-469a-adb9-0f490b4e4191","Type":"ContainerDied","Data":"c07115666dafe3d728c1df2aa6f88e9f9bbce1c7bb90c17c68d1fe6af563ab6d"} Sep 11 13:57:59.062000 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:59.061980 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fkrdxt" Sep 11 13:57:59.062000 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:57:59.062008 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c07115666dafe3d728c1df2aa6f88e9f9bbce1c7bb90c17c68d1fe6af563ab6d" Sep 11 13:58:00.018234 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.018202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:00.999425 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999395 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fgfdv"] Sep 11 13:58:00.999881 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999852 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="extract" Sep 11 13:58:00.999881 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999882 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="extract" Sep 11 13:58:01.000053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999903 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="pull" Sep 11 13:58:01.000053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999911 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="pull" Sep 11 13:58:01.000053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999941 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="util" Sep 11 13:58:01.000053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:00.999949 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="util" Sep 11 13:58:01.000053 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.000028 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db1594fe-1404-469a-adb9-0f490b4e4191" containerName="extract" Sep 11 13:58:01.003864 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.003841 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.006376 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.006354 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-qxmm7\"" Sep 11 13:58:01.015418 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.015400 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fgfdv"] Sep 11 13:58:01.102568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.102539 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/198c9009-2ad3-4195-a8f8-c1f99a022163-bound-sa-token\") pod \"cert-manager-55b4d954d5-fgfdv\" (UID: \"198c9009-2ad3-4195-a8f8-c1f99a022163\") " pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.102983 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.102580 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnbt2\" (UniqueName: \"kubernetes.io/projected/198c9009-2ad3-4195-a8f8-c1f99a022163-kube-api-access-tnbt2\") pod \"cert-manager-55b4d954d5-fgfdv\" (UID: \"198c9009-2ad3-4195-a8f8-c1f99a022163\") " pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.203887 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.203854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnbt2\" (UniqueName: \"kubernetes.io/projected/198c9009-2ad3-4195-a8f8-c1f99a022163-kube-api-access-tnbt2\") pod \"cert-manager-55b4d954d5-fgfdv\" (UID: \"198c9009-2ad3-4195-a8f8-c1f99a022163\") " pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.204073 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.203982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/198c9009-2ad3-4195-a8f8-c1f99a022163-bound-sa-token\") pod \"cert-manager-55b4d954d5-fgfdv\" (UID: \"198c9009-2ad3-4195-a8f8-c1f99a022163\") " pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.212998 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.212968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/198c9009-2ad3-4195-a8f8-c1f99a022163-bound-sa-token\") pod \"cert-manager-55b4d954d5-fgfdv\" (UID: \"198c9009-2ad3-4195-a8f8-c1f99a022163\") " pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.213172 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.213147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnbt2\" (UniqueName: \"kubernetes.io/projected/198c9009-2ad3-4195-a8f8-c1f99a022163-kube-api-access-tnbt2\") pod \"cert-manager-55b4d954d5-fgfdv\" (UID: \"198c9009-2ad3-4195-a8f8-c1f99a022163\") " pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.315307 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.315271 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fgfdv" Sep 11 13:58:01.895458 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:01.895438 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fgfdv"] Sep 11 13:58:01.897748 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:58:01.897728 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod198c9009_2ad3_4195_a8f8_c1f99a022163.slice/crio-d7b7b030081d6c637120d6f42813e50696891f08efe4d0ba0b727edeb6126e98 WatchSource:0}: Error finding container d7b7b030081d6c637120d6f42813e50696891f08efe4d0ba0b727edeb6126e98: Status 404 returned error can't find the container with id d7b7b030081d6c637120d6f42813e50696891f08efe4d0ba0b727edeb6126e98 Sep 11 13:58:02.079149 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:02.079091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" event={"ID":"f1487f96-5a51-41e8-845f-728e6cce893a","Type":"ContainerStarted","Data":"938c3a73b02f05d10765749f5be10c8535351b0fff942831eee45ad5df880a50"} Sep 11 13:58:02.079336 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:02.079185 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:58:02.080694 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:02.080668 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fgfdv" event={"ID":"198c9009-2ad3-4195-a8f8-c1f99a022163","Type":"ContainerStarted","Data":"b1b31de842118dcbc4d555a325b2352c36493f1c2bda50622451202c5c75501c"} Sep 11 13:58:02.080804 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:02.080701 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fgfdv" event={"ID":"198c9009-2ad3-4195-a8f8-c1f99a022163","Type":"ContainerStarted","Data":"d7b7b030081d6c637120d6f42813e50696891f08efe4d0ba0b727edeb6126e98"} Sep 11 13:58:02.098470 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:02.098403 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" podStartSLOduration=1.373430114 podStartE2EDuration="11.098387634s" totalImagesPullingTime="9.72495752s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:57:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:02.095962353 +0000 UTC m=+298.965693614" watchObservedRunningTime="2026-09-11 13:58:02.098387634 +0000 UTC m=+298.968118896" Sep 11 13:58:02.112589 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:02.112546 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fgfdv" podStartSLOduration=2.112531482 podStartE2EDuration="2.112531482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:02.111443838 +0000 UTC m=+298.981175117" watchObservedRunningTime="2026-09-11 13:58:02.112531482 +0000 UTC m=+298.982262743" Sep 11 13:58:03.165120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:03.165074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:03.265619 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:03.265583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:03.563569 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:03.563544 2579 kubelet.go:1648] "Image garbage collection succeeded" Sep 11 13:58:08.087629 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:08.087594 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-7xzjc" Sep 11 13:58:09.279783 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:09.279738 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:09.382282 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:09.382239 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:11.004877 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.004841 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs"] Sep 11 13:58:11.007269 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.007251 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.009764 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.009746 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 13:58:11.010038 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.010024 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 13:58:11.010222 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.010210 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29475\"" Sep 11 13:58:11.018290 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.018271 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs"] Sep 11 13:58:11.083486 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.083460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww4kx\" (UniqueName: \"kubernetes.io/projected/9e3c0fcf-f748-4817-b8d5-4be306442b84-kube-api-access-ww4kx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.083592 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.083491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.083592 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.083516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.184801 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.184772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww4kx\" (UniqueName: \"kubernetes.io/projected/9e3c0fcf-f748-4817-b8d5-4be306442b84-kube-api-access-ww4kx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.184923 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.184809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.184923 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.184837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.185192 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.185174 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.185235 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.185204 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.194028 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.194000 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww4kx\" (UniqueName: \"kubernetes.io/projected/9e3c0fcf-f748-4817-b8d5-4be306442b84-kube-api-access-ww4kx\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.200152 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.200121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:11.316308 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.316282 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:11.591851 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:11.591825 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs"] Sep 11 13:58:11.593989 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:58:11.593962 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3c0fcf_f748_4817_b8d5_4be306442b84.slice/crio-82061d06f1783e2a5368598a1c26ee22644901a5d71c4fdf168243f5343603c5 WatchSource:0}: Error finding container 82061d06f1783e2a5368598a1c26ee22644901a5d71c4fdf168243f5343603c5: Status 404 returned error can't find the container with id 82061d06f1783e2a5368598a1c26ee22644901a5d71c4fdf168243f5343603c5 Sep 11 13:58:12.130986 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:12.130951 2579 generic.go:356] "Generic (PLEG): container finished" podID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerID="0825245c17daa93157914ceb7bb8b8697e2082d154b11a2a8a232037dcbcc218" exitCode=0 Sep 11 13:58:12.131396 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:12.131039 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" event={"ID":"9e3c0fcf-f748-4817-b8d5-4be306442b84","Type":"ContainerDied","Data":"0825245c17daa93157914ceb7bb8b8697e2082d154b11a2a8a232037dcbcc218"} Sep 11 13:58:12.131396 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:12.131090 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" event={"ID":"9e3c0fcf-f748-4817-b8d5-4be306442b84","Type":"ContainerStarted","Data":"82061d06f1783e2a5368598a1c26ee22644901a5d71c4fdf168243f5343603c5"} Sep 11 13:58:12.131541 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:12.131524 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 13:58:13.138064 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:13.138031 2579 generic.go:356] "Generic (PLEG): container finished" podID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerID="7d788cea67919172b486652984381c33fbc179150a4c530f4365cd6c574ec0af" exitCode=0 Sep 11 13:58:13.138482 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:13.138094 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" event={"ID":"9e3c0fcf-f748-4817-b8d5-4be306442b84","Type":"ContainerDied","Data":"7d788cea67919172b486652984381c33fbc179150a4c530f4365cd6c574ec0af"} Sep 11 13:58:14.145394 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:14.145356 2579 generic.go:356] "Generic (PLEG): container finished" podID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerID="4d911555ad920a43c545bd918b667c65744217a587dea3daef28bd959835433b" exitCode=0 Sep 11 13:58:14.145774 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:14.145420 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" event={"ID":"9e3c0fcf-f748-4817-b8d5-4be306442b84","Type":"ContainerDied","Data":"4d911555ad920a43c545bd918b667c65744217a587dea3daef28bd959835433b"} Sep 11 13:58:15.273716 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.273689 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:15.422096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.422017 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-util\") pod \"9e3c0fcf-f748-4817-b8d5-4be306442b84\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " Sep 11 13:58:15.422258 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.422123 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww4kx\" (UniqueName: \"kubernetes.io/projected/9e3c0fcf-f748-4817-b8d5-4be306442b84-kube-api-access-ww4kx\") pod \"9e3c0fcf-f748-4817-b8d5-4be306442b84\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " Sep 11 13:58:15.422258 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.422162 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-bundle\") pod \"9e3c0fcf-f748-4817-b8d5-4be306442b84\" (UID: \"9e3c0fcf-f748-4817-b8d5-4be306442b84\") " Sep 11 13:58:15.422805 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.422777 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-bundle" (OuterVolumeSpecName: "bundle") pod "9e3c0fcf-f748-4817-b8d5-4be306442b84" (UID: "9e3c0fcf-f748-4817-b8d5-4be306442b84"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:58:15.424171 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.424141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3c0fcf-f748-4817-b8d5-4be306442b84-kube-api-access-ww4kx" (OuterVolumeSpecName: "kube-api-access-ww4kx") pod "9e3c0fcf-f748-4817-b8d5-4be306442b84" (UID: "9e3c0fcf-f748-4817-b8d5-4be306442b84"). InnerVolumeSpecName "kube-api-access-ww4kx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:58:15.429867 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.429843 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-util" (OuterVolumeSpecName: "util") pod "9e3c0fcf-f748-4817-b8d5-4be306442b84" (UID: "9e3c0fcf-f748-4817-b8d5-4be306442b84"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:58:15.523831 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.523801 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:58:15.523831 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.523827 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww4kx\" (UniqueName: \"kubernetes.io/projected/9e3c0fcf-f748-4817-b8d5-4be306442b84-kube-api-access-ww4kx\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:58:15.523831 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:15.523838 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9e3c0fcf-f748-4817-b8d5-4be306442b84-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:58:16.163155 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:16.163095 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" event={"ID":"9e3c0fcf-f748-4817-b8d5-4be306442b84","Type":"ContainerDied","Data":"82061d06f1783e2a5368598a1c26ee22644901a5d71c4fdf168243f5343603c5"} Sep 11 13:58:16.163316 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:16.163193 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82061d06f1783e2a5368598a1c26ee22644901a5d71c4fdf168243f5343603c5" Sep 11 13:58:16.163316 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:16.163132 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af94rhs" Sep 11 13:58:22.060617 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060585 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g"] Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060939 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="pull" Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060950 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="pull" Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060971 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="util" Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060977 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="util" Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060990 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="extract" Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.060996 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="extract" Sep 11 13:58:22.061120 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.061050 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e3c0fcf-f748-4817-b8d5-4be306442b84" containerName="extract" Sep 11 13:58:22.063683 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.063665 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.068078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.068055 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 13:58:22.068231 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.068212 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29475\"" Sep 11 13:58:22.068303 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.068230 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 13:58:22.085311 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.085287 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g"] Sep 11 13:58:22.181682 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.181632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.181855 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.181751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.181855 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.181792 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb98k\" (UniqueName: \"kubernetes.io/projected/50fb1a97-e360-48e6-8fa7-be76d673bca9-kube-api-access-wb98k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.282437 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.282399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.282612 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.282444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb98k\" (UniqueName: \"kubernetes.io/projected/50fb1a97-e360-48e6-8fa7-be76d673bca9-kube-api-access-wb98k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.282612 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.282498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.282835 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.282813 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.282871 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.282830 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.302182 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.302147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb98k\" (UniqueName: \"kubernetes.io/projected/50fb1a97-e360-48e6-8fa7-be76d673bca9-kube-api-access-wb98k\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.373056 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.372976 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:22.712076 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.712043 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g"] Sep 11 13:58:22.712645 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:58:22.712621 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fb1a97_e360_48e6_8fa7_be76d673bca9.slice/crio-bb76812e1d9f74ec21e752b0742d9a5606ed9a5c9ad1b87bd4889398b6d612de WatchSource:0}: Error finding container bb76812e1d9f74ec21e752b0742d9a5606ed9a5c9ad1b87bd4889398b6d612de: Status 404 returned error can't find the container with id bb76812e1d9f74ec21e752b0742d9a5606ed9a5c9ad1b87bd4889398b6d612de Sep 11 13:58:22.825962 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.825927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:22.925787 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:22.925719 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:23.195388 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.195353 2579 generic.go:356] "Generic (PLEG): container finished" podID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerID="e9adf47da8f2ab7e7666076839c2a18c11081e2aa408a82272a5480c3c7d2559" exitCode=0 Sep 11 13:58:23.195778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.195433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" event={"ID":"50fb1a97-e360-48e6-8fa7-be76d673bca9","Type":"ContainerDied","Data":"e9adf47da8f2ab7e7666076839c2a18c11081e2aa408a82272a5480c3c7d2559"} Sep 11 13:58:23.195778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.195480 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" event={"ID":"50fb1a97-e360-48e6-8fa7-be76d673bca9","Type":"ContainerStarted","Data":"bb76812e1d9f74ec21e752b0742d9a5606ed9a5c9ad1b87bd4889398b6d612de"} Sep 11 13:58:23.286972 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.286936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:23.495676 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.495594 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:23.592802 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.592753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:23.688945 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.688916 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb"] Sep 11 13:58:23.691228 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.691212 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.695013 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.694983 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Sep 11 13:58:23.695143 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.695040 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Sep 11 13:58:23.695573 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.695556 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rkxmr\"" Sep 11 13:58:23.699851 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.699831 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Sep 11 13:58:23.700632 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.700459 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Sep 11 13:58:23.716554 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.716524 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb"] Sep 11 13:58:23.744514 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.744488 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:23.793428 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.793404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.793568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.793435 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw6xr\" (UniqueName: \"kubernetes.io/projected/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-kube-api-access-gw6xr\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.793568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.793460 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.793568 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.793498 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-tmp\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.793731 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.793574 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-manifests\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.895001 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.894968 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.895001 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.895007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw6xr\" (UniqueName: \"kubernetes.io/projected/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-kube-api-access-gw6xr\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.895250 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.895031 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.895250 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.895048 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-tmp\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.895250 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.895090 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-manifests\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.895588 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.895567 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-manifests\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.897574 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.897547 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-tmp\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.897694 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.897606 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.897694 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.897653 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.904395 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.904372 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw6xr\" (UniqueName: \"kubernetes.io/projected/c3787eaa-67b6-4214-8889-9c6aa4a15c6b-kube-api-access-gw6xr\") pod \"opendatahub-operator-controller-manager-7ff676d985-tmbdb\" (UID: \"c3787eaa-67b6-4214-8889-9c6aa4a15c6b\") " pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:23.957012 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:23.956981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:24.003851 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.003772 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:24.060165 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.060121 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:24.155635 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.155597 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:24.202360 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.202331 2579 generic.go:356] "Generic (PLEG): container finished" podID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerID="27e100522b60a67702eb35fbf9bed6981580a497c1b4dada24f49273dcbc25b9" exitCode=0 Sep 11 13:58:24.202763 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.202421 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" event={"ID":"50fb1a97-e360-48e6-8fa7-be76d673bca9","Type":"ContainerDied","Data":"27e100522b60a67702eb35fbf9bed6981580a497c1b4dada24f49273dcbc25b9"} Sep 11 13:58:24.229250 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.229219 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:24.280478 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.280446 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb"] Sep 11 13:58:24.301802 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.301771 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:24.402519 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.402478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:24.502608 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:24.502568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:25.211413 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:25.211341 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" event={"ID":"c3787eaa-67b6-4214-8889-9c6aa4a15c6b","Type":"ContainerStarted","Data":"b8e489d43dfd90b0987be1c1b80cb028e56f1504d39a8b2ab612534e7795fc9e"} Sep 11 13:58:25.214381 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:25.214345 2579 generic.go:356] "Generic (PLEG): container finished" podID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerID="1d581ddf724d98d71563911e953e0d15aa3910b19a134825f9db81129fb1c4aa" exitCode=0 Sep 11 13:58:25.214517 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:25.214389 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" event={"ID":"50fb1a97-e360-48e6-8fa7-be76d673bca9","Type":"ContainerDied","Data":"1d581ddf724d98d71563911e953e0d15aa3910b19a134825f9db81129fb1c4aa"} Sep 11 13:58:27.159958 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.159933 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:27.225939 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.225906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" event={"ID":"50fb1a97-e360-48e6-8fa7-be76d673bca9","Type":"ContainerDied","Data":"bb76812e1d9f74ec21e752b0742d9a5606ed9a5c9ad1b87bd4889398b6d612de"} Sep 11 13:58:27.226051 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.225960 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb76812e1d9f74ec21e752b0742d9a5606ed9a5c9ad1b87bd4889398b6d612de" Sep 11 13:58:27.226051 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.225919 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b04rg9g" Sep 11 13:58:27.327045 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.327008 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-bundle\") pod \"50fb1a97-e360-48e6-8fa7-be76d673bca9\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " Sep 11 13:58:27.327250 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.327062 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-util\") pod \"50fb1a97-e360-48e6-8fa7-be76d673bca9\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " Sep 11 13:58:27.327250 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.327139 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb98k\" (UniqueName: \"kubernetes.io/projected/50fb1a97-e360-48e6-8fa7-be76d673bca9-kube-api-access-wb98k\") pod \"50fb1a97-e360-48e6-8fa7-be76d673bca9\" (UID: \"50fb1a97-e360-48e6-8fa7-be76d673bca9\") " Sep 11 13:58:27.327863 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.327810 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-bundle" (OuterVolumeSpecName: "bundle") pod "50fb1a97-e360-48e6-8fa7-be76d673bca9" (UID: "50fb1a97-e360-48e6-8fa7-be76d673bca9"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:58:27.329309 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.329284 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50fb1a97-e360-48e6-8fa7-be76d673bca9-kube-api-access-wb98k" (OuterVolumeSpecName: "kube-api-access-wb98k") pod "50fb1a97-e360-48e6-8fa7-be76d673bca9" (UID: "50fb1a97-e360-48e6-8fa7-be76d673bca9"). InnerVolumeSpecName "kube-api-access-wb98k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 13:58:27.332441 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.332416 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-util" (OuterVolumeSpecName: "util") pod "50fb1a97-e360-48e6-8fa7-be76d673bca9" (UID: "50fb1a97-e360-48e6-8fa7-be76d673bca9"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 13:58:27.428729 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.428644 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:58:27.428729 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.428673 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/50fb1a97-e360-48e6-8fa7-be76d673bca9-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:58:27.428729 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:27.428682 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb98k\" (UniqueName: \"kubernetes.io/projected/50fb1a97-e360-48e6-8fa7-be76d673bca9-kube-api-access-wb98k\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 13:58:28.232000 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:28.231969 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" event={"ID":"c3787eaa-67b6-4214-8889-9c6aa4a15c6b","Type":"ContainerStarted","Data":"74453d91af31a4df5050ac06181391c02ddaff7e7e5a653ccd8495f9d03b6884"} Sep 11 13:58:29.238504 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:29.238466 2579 generic.go:356] "Generic (PLEG): container finished" podID="c3787eaa-67b6-4214-8889-9c6aa4a15c6b" containerID="74453d91af31a4df5050ac06181391c02ddaff7e7e5a653ccd8495f9d03b6884" exitCode=0 Sep 11 13:58:29.238916 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:29.238554 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" event={"ID":"c3787eaa-67b6-4214-8889-9c6aa4a15c6b","Type":"ContainerDied","Data":"74453d91af31a4df5050ac06181391c02ddaff7e7e5a653ccd8495f9d03b6884"} Sep 11 13:58:30.245558 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:30.245517 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" event={"ID":"c3787eaa-67b6-4214-8889-9c6aa4a15c6b","Type":"ContainerStarted","Data":"a1edd7cbf3b889d082f219b6129b91ae848ac8566086f886f50658411b8983ed"} Sep 11 13:58:30.245922 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:30.245667 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:30.271645 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:30.271598 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" podStartSLOduration=3.06874745 podStartE2EDuration="7.271583343s" totalImagesPullingTime="3.19280787s" totalInitContainerRuntime="1.010028023s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:30.269701149 +0000 UTC m=+327.139432411" watchObservedRunningTime="2026-09-11 13:58:30.271583343 +0000 UTC m=+327.141314643" Sep 11 13:58:31.491013 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:31.490972 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:32.710422 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:32.710384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:41.253256 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:41.253220 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7ff676d985-tmbdb" Sep 11 13:58:43.034476 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.034440 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:43.156245 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.156213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:43.591195 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.591157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:43.645834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.645807 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp"] Sep 11 13:58:43.646129 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646117 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="util" Sep 11 13:58:43.646181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646132 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="util" Sep 11 13:58:43.646181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646149 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="pull" Sep 11 13:58:43.646181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646154 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="pull" Sep 11 13:58:43.646181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646159 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="extract" Sep 11 13:58:43.646181 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646164 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="extract" Sep 11 13:58:43.646331 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.646221 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="50fb1a97-e360-48e6-8fa7-be76d673bca9" containerName="extract" Sep 11 13:58:43.654650 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.654627 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.657636 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.657616 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Sep 11 13:58:43.658291 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.658273 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Sep 11 13:58:43.667316 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.667289 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp"] Sep 11 13:58:43.670974 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.670951 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Sep 11 13:58:43.671285 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.671266 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Sep 11 13:58:43.755930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.755897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcn7f\" (UniqueName: \"kubernetes.io/projected/f3137645-bb52-4763-905c-70b1f787077f-kube-api-access-jcn7f\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.756096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.755939 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3137645-bb52-4763-905c-70b1f787077f-tmp\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.756096 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.756006 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3137645-bb52-4763-905c-70b1f787077f-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.856610 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.856521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3137645-bb52-4763-905c-70b1f787077f-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.856610 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:58:43.856596 2579 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Sep 11 13:58:43.856835 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.856615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcn7f\" (UniqueName: \"kubernetes.io/projected/f3137645-bb52-4763-905c-70b1f787077f-kube-api-access-jcn7f\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.856835 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.856642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3137645-bb52-4763-905c-70b1f787077f-tmp\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.856835 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:58:43.856655 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f3137645-bb52-4763-905c-70b1f787077f-tls-certs podName:f3137645-bb52-4763-905c-70b1f787077f nodeName:}" failed. No retries permitted until 2026-09-11 13:58:44.356639309 +0000 UTC m=+341.226370549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f3137645-bb52-4763-905c-70b1f787077f-tls-certs") pod "kube-auth-proxy-746c8b5fff-js2tp" (UID: "f3137645-bb52-4763-905c-70b1f787077f") : secret "kube-auth-proxy-tls" not found Sep 11 13:58:43.858968 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.858949 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3137645-bb52-4763-905c-70b1f787077f-tmp\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:43.877050 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:43.877021 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcn7f\" (UniqueName: \"kubernetes.io/projected/f3137645-bb52-4763-905c-70b1f787077f-kube-api-access-jcn7f\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:44.360857 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:44.360826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3137645-bb52-4763-905c-70b1f787077f-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:44.363303 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:44.363287 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3137645-bb52-4763-905c-70b1f787077f-tls-certs\") pod \"kube-auth-proxy-746c8b5fff-js2tp\" (UID: \"f3137645-bb52-4763-905c-70b1f787077f\") " pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:44.463894 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:44.463860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:44.563930 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:44.563898 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" Sep 11 13:58:44.845131 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:44.845074 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp"] Sep 11 13:58:44.849063 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:58:44.849032 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3137645_bb52_4763_905c_70b1f787077f.slice/crio-6a64a62adee40984a356b2470a20f15a71fb7f4b49755bfc085bb8116b9a013e WatchSource:0}: Error finding container 6a64a62adee40984a356b2470a20f15a71fb7f4b49755bfc085bb8116b9a013e: Status 404 returned error can't find the container with id 6a64a62adee40984a356b2470a20f15a71fb7f4b49755bfc085bb8116b9a013e Sep 11 13:58:45.091974 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:45.091943 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:45.241319 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:45.241281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:45.316243 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:45.316212 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" event={"ID":"f3137645-bb52-4763-905c-70b1f787077f","Type":"ContainerStarted","Data":"6a64a62adee40984a356b2470a20f15a71fb7f4b49755bfc085bb8116b9a013e"} Sep 11 13:58:46.323292 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:46.323195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:48.334536 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:48.334506 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" event={"ID":"f3137645-bb52-4763-905c-70b1f787077f","Type":"ContainerStarted","Data":"1a6a6526fdf7d89da4b57aadb5344a88d8f8be1b8b170fc2394bc83ee8f794d7"} Sep 11 13:58:48.355050 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:48.354990 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-746c8b5fff-js2tp" podStartSLOduration=2.596977161 podStartE2EDuration="5.354969463s" totalImagesPullingTime="2.757992302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:48.35232666 +0000 UTC m=+345.222057924" watchObservedRunningTime="2026-09-11 13:58:48.354969463 +0000 UTC m=+345.224700726" Sep 11 13:58:50.265730 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.265690 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:50.366629 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.366591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:50.919071 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.919038 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5"] Sep 11 13:58:50.922704 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.922682 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:50.925658 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.925636 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Sep 11 13:58:50.925791 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.925720 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-n9pjz\"" Sep 11 13:58:50.933622 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.933602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5"] Sep 11 13:58:50.961010 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:50.960969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:51.019865 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.019832 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.019880 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.019898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.019917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzpc\" (UniqueName: \"kubernetes.io/projected/c7405e28-03ad-4b1f-b1b8-a120c901998c-kube-api-access-bwzpc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.019968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020023 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.020005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020254 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.020054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020254 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.020086 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7405e28-03ad-4b1f-b1b8-a120c901998c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.020254 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.020123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.120808 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120774 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.120808 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7405e28-03ad-4b1f-b1b8-a120c901998c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120917 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120940 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.120967 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzpc\" (UniqueName: \"kubernetes.io/projected/c7405e28-03ad-4b1f-b1b8-a120c901998c-kube-api-access-bwzpc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121039 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121403 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121403 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121516 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121472 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121572 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121553 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121615 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121575 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.121651 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.121636 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7405e28-03ad-4b1f-b1b8-a120c901998c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.123480 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.123456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.123721 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.123704 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.139150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.139127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7405e28-03ad-4b1f-b1b8-a120c901998c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.139304 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.139282 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzpc\" (UniqueName: \"kubernetes.io/projected/c7405e28-03ad-4b1f-b1b8-a120c901998c-kube-api-access-bwzpc\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5\" (UID: \"c7405e28-03ad-4b1f-b1b8-a120c901998c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.168546 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.168502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:51.234065 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.233990 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:51.261379 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.261337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:51.517804 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:51.514087 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5"] Sep 11 13:58:52.355690 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:52.355653 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" event={"ID":"c7405e28-03ad-4b1f-b1b8-a120c901998c","Type":"ContainerStarted","Data":"7ff8da41ba4d6dd9bfc7a903757afa641826b71ea7dcf510da8d79e5372db5b6"} Sep 11 13:58:54.104463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.104186 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7zncc"] Sep 11 13:58:54.109706 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.109676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.113459 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.113431 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-bhqn2\"" Sep 11 13:58:54.113712 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.113691 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Sep 11 13:58:54.117810 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.117785 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7zncc"] Sep 11 13:58:54.138807 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.138769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:54.146496 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.146464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1241f51c-d8ea-4b01-8784-1f88bd5be660-cert\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.146651 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.146511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zt88m\" (UniqueName: \"kubernetes.io/projected/1241f51c-d8ea-4b01-8784-1f88bd5be660-kube-api-access-zt88m\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.243409 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.243369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:54.248246 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.248209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1241f51c-d8ea-4b01-8784-1f88bd5be660-cert\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.248387 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.248261 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zt88m\" (UniqueName: \"kubernetes.io/projected/1241f51c-d8ea-4b01-8784-1f88bd5be660-kube-api-access-zt88m\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.248387 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:58:54.248335 2579 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Sep 11 13:58:54.248501 ip-10-0-136-162 kubenswrapper[2579]: E0911 13:58:54.248436 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1241f51c-d8ea-4b01-8784-1f88bd5be660-cert podName:1241f51c-d8ea-4b01-8784-1f88bd5be660 nodeName:}" failed. No retries permitted until 2026-09-11 13:58:54.748407864 +0000 UTC m=+351.618139115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/1241f51c-d8ea-4b01-8784-1f88bd5be660-cert") pod "odh-model-controller-7998c67dc4-7zncc" (UID: "1241f51c-d8ea-4b01-8784-1f88bd5be660") : secret "odh-model-controller-webhook-cert" not found Sep 11 13:58:54.261707 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.261675 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zt88m\" (UniqueName: \"kubernetes.io/projected/1241f51c-d8ea-4b01-8784-1f88bd5be660-kube-api-access-zt88m\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.379179 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.379060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:54.479975 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.479940 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:54.754060 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.753943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1241f51c-d8ea-4b01-8784-1f88bd5be660-cert\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.757005 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.756978 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1241f51c-d8ea-4b01-8784-1f88bd5be660-cert\") pod \"odh-model-controller-7998c67dc4-7zncc\" (UID: \"1241f51c-d8ea-4b01-8784-1f88bd5be660\") " pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:54.779926 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:54.779884 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:55.011986 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:55.011895 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:58:55.027939 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:55.027887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:55.985435 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:55.985398 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 13:58:55.985797 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:55.985493 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 13:58:55.985797 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:55.985540 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 13:58:56.178213 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:56.178188 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-7zncc"] Sep 11 13:58:56.180592 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:58:56.180553 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1241f51c_d8ea_4b01_8784_1f88bd5be660.slice/crio-6793a07c03aa591fe0e8cfb001022d51eee5d5cb939035fb0fc84f574a13112d WatchSource:0}: Error finding container 6793a07c03aa591fe0e8cfb001022d51eee5d5cb939035fb0fc84f574a13112d: Status 404 returned error can't find the container with id 6793a07c03aa591fe0e8cfb001022d51eee5d5cb939035fb0fc84f574a13112d Sep 11 13:58:56.383287 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:56.383249 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" event={"ID":"c7405e28-03ad-4b1f-b1b8-a120c901998c","Type":"ContainerStarted","Data":"30562b01362b731bc3961dc1d994bc639183c98d13fdacec8f6af1c717483426"} Sep 11 13:58:56.384629 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:56.384608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" event={"ID":"1241f51c-d8ea-4b01-8784-1f88bd5be660","Type":"ContainerStarted","Data":"6793a07c03aa591fe0e8cfb001022d51eee5d5cb939035fb0fc84f574a13112d"} Sep 11 13:58:56.410738 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:56.410695 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" podStartSLOduration=1.944889639 podStartE2EDuration="6.410681969s" totalImagesPullingTime="4.46579233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:56.410522061 +0000 UTC m=+353.280253322" watchObservedRunningTime="2026-09-11 13:58:56.410681969 +0000 UTC m=+353.280413230" Sep 11 13:58:57.235254 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:57.235216 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:57.241617 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:57.241376 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:57.392754 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:57.392704 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:57.394248 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:57.394067 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5" Sep 11 13:58:59.404737 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:59.404698 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" event={"ID":"1241f51c-d8ea-4b01-8784-1f88bd5be660","Type":"ContainerStarted","Data":"ab3ebe320616f58b39abd777e559cf711f11813d2e221d9996962c4701580893"} Sep 11 13:58:59.405311 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:59.404886 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:58:59.434454 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:58:59.434404 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" podStartSLOduration=2.599556826 podStartE2EDuration="5.434388338s" totalImagesPullingTime="2.834831512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:58:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:59.432453301 +0000 UTC m=+356.302184557" watchObservedRunningTime="2026-09-11 13:58:59.434388338 +0000 UTC m=+356.304119600" Sep 11 13:59:00.872254 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.872218 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv"] Sep 11 13:59:00.875931 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.875910 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:00.878643 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.878623 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Sep 11 13:59:00.878782 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.878660 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Sep 11 13:59:00.878782 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.878749 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Sep 11 13:59:00.878911 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.878784 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-8pprv\"" Sep 11 13:59:00.895452 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:00.895432 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv"] Sep 11 13:59:01.010750 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.010719 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e32aa0-f398-4614-a389-7ce7913086eb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.010917 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.010781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wg7w\" (UniqueName: \"kubernetes.io/projected/79e32aa0-f398-4614-a389-7ce7913086eb-kube-api-access-7wg7w\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.010917 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.010855 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e32aa0-f398-4614-a389-7ce7913086eb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.011011 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.010930 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.011011 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.010958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.011011 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.011004 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.011178 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.011031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79e32aa0-f398-4614-a389-7ce7913086eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.111649 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.111592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.111848 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.111713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.111927 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.111866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.111995 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.111970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79e32aa0-f398-4614-a389-7ce7913086eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.112057 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.112022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e32aa0-f398-4614-a389-7ce7913086eb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.112146 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.112095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wg7w\" (UniqueName: \"kubernetes.io/projected/79e32aa0-f398-4614-a389-7ce7913086eb-kube-api-access-7wg7w\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.112208 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.112156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e32aa0-f398-4614-a389-7ce7913086eb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.112950 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.112917 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/79e32aa0-f398-4614-a389-7ce7913086eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.114709 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.114629 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.114839 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.114723 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.114839 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.114821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e32aa0-f398-4614-a389-7ce7913086eb-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.114955 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.114890 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/79e32aa0-f398-4614-a389-7ce7913086eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.121172 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.121135 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e32aa0-f398-4614-a389-7ce7913086eb-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.121284 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.121264 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wg7w\" (UniqueName: \"kubernetes.io/projected/79e32aa0-f398-4614-a389-7ce7913086eb-kube-api-access-7wg7w\") pod \"istiod-openshift-gateway-5677ccb49b-j97pv\" (UID: \"79e32aa0-f398-4614-a389-7ce7913086eb\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.185286 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.185221 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:01.465672 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:01.465644 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv"] Sep 11 13:59:01.467862 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:59:01.467835 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e32aa0_f398_4614_a389_7ce7913086eb.slice/crio-de12df500d83b1a142b5a51f98032cd51979c62b4049c2a3ab80814b8fb0cfdf WatchSource:0}: Error finding container de12df500d83b1a142b5a51f98032cd51979c62b4049c2a3ab80814b8fb0cfdf: Status 404 returned error can't find the container with id de12df500d83b1a142b5a51f98032cd51979c62b4049c2a3ab80814b8fb0cfdf Sep 11 13:59:02.428323 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:02.428285 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" event={"ID":"79e32aa0-f398-4614-a389-7ce7913086eb","Type":"ContainerStarted","Data":"de12df500d83b1a142b5a51f98032cd51979c62b4049c2a3ab80814b8fb0cfdf"} Sep 11 13:59:04.048323 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:04.048290 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 13:59:04.048583 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:04.048356 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 13:59:04.440956 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:04.440868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" event={"ID":"79e32aa0-f398-4614-a389-7ce7913086eb","Type":"ContainerStarted","Data":"44b40a30788982a07110e82add8850044266302b4ed922e6fcc16bbe6e6c05ef"} Sep 11 13:59:04.440956 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:04.440950 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:04.565909 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:04.565856 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" podStartSLOduration=1.988112472 podStartE2EDuration="4.565837324s" totalImagesPullingTime="2.577724852s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:59:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:59:04.55274754 +0000 UTC m=+361.422478802" watchObservedRunningTime="2026-09-11 13:59:04.565837324 +0000 UTC m=+361.435568586" Sep 11 13:59:05.447452 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:05.447425 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-j97pv" Sep 11 13:59:05.802547 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:05.802510 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.039463 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:06.039428 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.190421 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:06.190333 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.322333 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:06.322294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.439161 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:06.439124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:06.566028 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:06.565993 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:10.411827 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:10.411801 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-7zncc" Sep 11 13:59:42.287201 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:42.287165 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:44.502595 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:44.502559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:44.663258 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:44.663212 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:55.080921 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:55.080883 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:55.224353 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:55.224320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 13:59:58.731150 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.731121 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96"] Sep 11 13:59:58.733834 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.733815 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.736325 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.736301 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Sep 11 13:59:58.737049 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.737036 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Sep 11 13:59:58.737149 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.737084 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-29475\"" Sep 11 13:59:58.748333 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.748313 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96"] Sep 11 13:59:58.789165 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.789142 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5vg\" (UniqueName: \"kubernetes.io/projected/43a5de0d-249b-4e80-a955-01133ca9a3ca-kube-api-access-lz5vg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.789268 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.789178 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.789268 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.789196 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.819912 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.819889 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd"] Sep 11 13:59:58.822224 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.822210 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.832590 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.832571 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd"] Sep 11 13:59:58.889889 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.889868 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5vg\" (UniqueName: \"kubernetes.io/projected/43a5de0d-249b-4e80-a955-01133ca9a3ca-kube-api-access-lz5vg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.889989 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.889903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.889989 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.889966 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.890078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.890049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.890078 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.890073 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.890202 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.890097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlv8m\" (UniqueName: \"kubernetes.io/projected/497fb332-ef1f-488e-8865-953fad0af090-kube-api-access-jlv8m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.890452 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.890434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.890514 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.890448 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.900973 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.900951 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5vg\" (UniqueName: \"kubernetes.io/projected/43a5de0d-249b-4e80-a955-01133ca9a3ca-kube-api-access-lz5vg\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:58.923558 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.923535 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br"] Sep 11 13:59:58.925846 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.925832 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:58.936924 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.936903 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br"] Sep 11 13:59:58.990643 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.990591 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fpz2\" (UniqueName: \"kubernetes.io/projected/160460e1-29a3-4b35-adc9-9c6174ba0b6e-kube-api-access-4fpz2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:58.990643 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.990634 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:58.990778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.990731 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.990778 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.990763 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:58.990857 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.990785 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.990857 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.990811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlv8m\" (UniqueName: \"kubernetes.io/projected/497fb332-ef1f-488e-8865-953fad0af090-kube-api-access-jlv8m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.991050 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.991034 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:58.991097 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:58.991081 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:59.009577 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.009557 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlv8m\" (UniqueName: \"kubernetes.io/projected/497fb332-ef1f-488e-8865-953fad0af090-kube-api-access-jlv8m\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:59.043490 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.043474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 13:59:59.071685 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.071656 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm"] Sep 11 13:59:59.074358 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.074337 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.093151 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.092191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.093151 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.092269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fpz2\" (UniqueName: \"kubernetes.io/projected/160460e1-29a3-4b35-adc9-9c6174ba0b6e-kube-api-access-4fpz2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.093151 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.092334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.093151 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.092751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.093151 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.092997 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.099084 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.099056 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm"] Sep 11 13:59:59.112013 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.111991 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fpz2\" (UniqueName: \"kubernetes.io/projected/160460e1-29a3-4b35-adc9-9c6174ba0b6e-kube-api-access-4fpz2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.130853 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.130828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 13:59:59.193799 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.193759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.194033 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.194010 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.194148 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.194046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvsbd\" (UniqueName: \"kubernetes.io/projected/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-kube-api-access-tvsbd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.234594 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.234573 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 13:59:59.295631 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.295596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.295757 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.295637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvsbd\" (UniqueName: \"kubernetes.io/projected/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-kube-api-access-tvsbd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.295815 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.295773 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.296019 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.295998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.296159 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.296142 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.305238 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.305193 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvsbd\" (UniqueName: \"kubernetes.io/projected/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-kube-api-access-tvsbd\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.320494 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.320471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96"] Sep 11 13:59:59.322067 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:59:59.322045 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43a5de0d_249b_4e80_a955_01133ca9a3ca.slice/crio-50fe781d1f9c9d86bc43e02ead18963ec364f9c75fde222cf4a9211043d8e4d9 WatchSource:0}: Error finding container 50fe781d1f9c9d86bc43e02ead18963ec364f9c75fde222cf4a9211043d8e4d9: Status 404 returned error can't find the container with id 50fe781d1f9c9d86bc43e02ead18963ec364f9c75fde222cf4a9211043d8e4d9 Sep 11 13:59:59.385227 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.385206 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 13:59:59.404740 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.404711 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd"] Sep 11 13:59:59.405403 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:59:59.405372 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497fb332_ef1f_488e_8865_953fad0af090.slice/crio-e0a3dd949005a602f065ffe12879b3a9a1c074fa76249f783e3979c742753ffe WatchSource:0}: Error finding container e0a3dd949005a602f065ffe12879b3a9a1c074fa76249f783e3979c742753ffe: Status 404 returned error can't find the container with id e0a3dd949005a602f065ffe12879b3a9a1c074fa76249f783e3979c742753ffe Sep 11 13:59:59.507686 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.507664 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br"] Sep 11 13:59:59.509223 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:59:59.509197 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160460e1_29a3_4b35_adc9_9c6174ba0b6e.slice/crio-c888e038456b4e1d1fd1d92935f71d0aec2e8f25f474cb3553080fd40a41ba2a WatchSource:0}: Error finding container c888e038456b4e1d1fd1d92935f71d0aec2e8f25f474cb3553080fd40a41ba2a: Status 404 returned error can't find the container with id c888e038456b4e1d1fd1d92935f71d0aec2e8f25f474cb3553080fd40a41ba2a Sep 11 13:59:59.662763 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.662736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm"] Sep 11 13:59:59.676824 ip-10-0-136-162 kubenswrapper[2579]: W0911 13:59:59.676798 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda82aa63a_659a_41a0_acb3_1d10fbc3f5a6.slice/crio-f87e45bf21fa2a3eae829ef31db00d02ab94d30b841b75086a784061a6533513 WatchSource:0}: Error finding container f87e45bf21fa2a3eae829ef31db00d02ab94d30b841b75086a784061a6533513: Status 404 returned error can't find the container with id f87e45bf21fa2a3eae829ef31db00d02ab94d30b841b75086a784061a6533513 Sep 11 13:59:59.713794 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.713772 2579 generic.go:356] "Generic (PLEG): container finished" podID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerID="ad4e9abdec80da277e6a04460e3ad483c40e323e99f6ed6847f69f7731f6097f" exitCode=0 Sep 11 13:59:59.713890 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.713839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" event={"ID":"160460e1-29a3-4b35-adc9-9c6174ba0b6e","Type":"ContainerDied","Data":"ad4e9abdec80da277e6a04460e3ad483c40e323e99f6ed6847f69f7731f6097f"} Sep 11 13:59:59.713946 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.713905 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" event={"ID":"160460e1-29a3-4b35-adc9-9c6174ba0b6e","Type":"ContainerStarted","Data":"c888e038456b4e1d1fd1d92935f71d0aec2e8f25f474cb3553080fd40a41ba2a"} Sep 11 13:59:59.715702 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.715668 2579 generic.go:356] "Generic (PLEG): container finished" podID="497fb332-ef1f-488e-8865-953fad0af090" containerID="c836ecb94244d5bb4a00fafccf7efc2fa90b5d1a20083de634bab50977e793a7" exitCode=0 Sep 11 13:59:59.715811 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.715794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" event={"ID":"497fb332-ef1f-488e-8865-953fad0af090","Type":"ContainerDied","Data":"c836ecb94244d5bb4a00fafccf7efc2fa90b5d1a20083de634bab50977e793a7"} Sep 11 13:59:59.715892 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.715868 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" event={"ID":"497fb332-ef1f-488e-8865-953fad0af090","Type":"ContainerStarted","Data":"e0a3dd949005a602f065ffe12879b3a9a1c074fa76249f783e3979c742753ffe"} Sep 11 13:59:59.717355 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.717325 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" event={"ID":"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6","Type":"ContainerStarted","Data":"f87e45bf21fa2a3eae829ef31db00d02ab94d30b841b75086a784061a6533513"} Sep 11 13:59:59.719263 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.719241 2579 generic.go:356] "Generic (PLEG): container finished" podID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerID="ee4178459fe394f1afd587d120cc61b95493b2ccc56bd94290c85d925580f985" exitCode=0 Sep 11 13:59:59.719359 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.719311 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" event={"ID":"43a5de0d-249b-4e80-a955-01133ca9a3ca","Type":"ContainerDied","Data":"ee4178459fe394f1afd587d120cc61b95493b2ccc56bd94290c85d925580f985"} Sep 11 13:59:59.719359 ip-10-0-136-162 kubenswrapper[2579]: I0911 13:59:59.719343 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" event={"ID":"43a5de0d-249b-4e80-a955-01133ca9a3ca","Type":"ContainerStarted","Data":"50fe781d1f9c9d86bc43e02ead18963ec364f9c75fde222cf4a9211043d8e4d9"} Sep 11 14:00:00.063839 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.063808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:00.725905 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.725879 2579 generic.go:356] "Generic (PLEG): container finished" podID="497fb332-ef1f-488e-8865-953fad0af090" containerID="dd42200e10a1469e45195ca93081c5ddb3a5f4fc8ab0059a0636cd2876ab1eb5" exitCode=0 Sep 11 14:00:00.726034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.725914 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" event={"ID":"497fb332-ef1f-488e-8865-953fad0af090","Type":"ContainerDied","Data":"dd42200e10a1469e45195ca93081c5ddb3a5f4fc8ab0059a0636cd2876ab1eb5"} Sep 11 14:00:00.727711 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.727692 2579 generic.go:356] "Generic (PLEG): container finished" podID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerID="93362ea018b292951f6b53efc5eefaabfe69044c8cf97d78028245477045f42c" exitCode=0 Sep 11 14:00:00.727810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.727779 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" event={"ID":"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6","Type":"ContainerDied","Data":"93362ea018b292951f6b53efc5eefaabfe69044c8cf97d78028245477045f42c"} Sep 11 14:00:00.730018 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.729919 2579 generic.go:356] "Generic (PLEG): container finished" podID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerID="4edeba7d5c3fb86e67462bab50fb38f06790bafc267ed3908aff9a12879aad5c" exitCode=0 Sep 11 14:00:00.730018 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:00.729964 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" event={"ID":"43a5de0d-249b-4e80-a955-01133ca9a3ca","Type":"ContainerDied","Data":"4edeba7d5c3fb86e67462bab50fb38f06790bafc267ed3908aff9a12879aad5c"} Sep 11 14:00:01.737895 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:01.737856 2579 generic.go:356] "Generic (PLEG): container finished" podID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerID="5b26be2666aeb8260c8ea730814a8111f7c41a0f1b34c0f45032ee17267ba0cb" exitCode=0 Sep 11 14:00:01.738364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:01.737928 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" event={"ID":"43a5de0d-249b-4e80-a955-01133ca9a3ca","Type":"ContainerDied","Data":"5b26be2666aeb8260c8ea730814a8111f7c41a0f1b34c0f45032ee17267ba0cb"} Sep 11 14:00:01.739854 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:01.739827 2579 generic.go:356] "Generic (PLEG): container finished" podID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerID="bc81591938587d5d282e13cb8504d860e0c39dd4a79c03b98ea9472166c10519" exitCode=0 Sep 11 14:00:01.739992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:01.739910 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" event={"ID":"160460e1-29a3-4b35-adc9-9c6174ba0b6e","Type":"ContainerDied","Data":"bc81591938587d5d282e13cb8504d860e0c39dd4a79c03b98ea9472166c10519"} Sep 11 14:00:01.742351 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:01.742333 2579 generic.go:356] "Generic (PLEG): container finished" podID="497fb332-ef1f-488e-8865-953fad0af090" containerID="1944d53b23a5f9e8677751be9e382d0a35ecdf0d523af52b07072f35ca4d0738" exitCode=0 Sep 11 14:00:01.742461 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:01.742392 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" event={"ID":"497fb332-ef1f-488e-8865-953fad0af090","Type":"ContainerDied","Data":"1944d53b23a5f9e8677751be9e382d0a35ecdf0d523af52b07072f35ca4d0738"} Sep 11 14:00:01.883917 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:01.883884 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160460e1_29a3_4b35_adc9_9c6174ba0b6e.slice/crio-b2e4050779ab2931ab299f3b0d35f6771db4ad3083047a5be9c2d7ce582df077.scope\": RecentStats: unable to find data in memory cache]" Sep 11 14:00:01.884069 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:01.883973 2579 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod160460e1_29a3_4b35_adc9_9c6174ba0b6e.slice/crio-b2e4050779ab2931ab299f3b0d35f6771db4ad3083047a5be9c2d7ce582df077.scope\": RecentStats: unable to find data in memory cache]" Sep 11 14:00:02.750114 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:02.750062 2579 generic.go:356] "Generic (PLEG): container finished" podID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerID="b2e4050779ab2931ab299f3b0d35f6771db4ad3083047a5be9c2d7ce582df077" exitCode=0 Sep 11 14:00:02.750475 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:02.750142 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" event={"ID":"160460e1-29a3-4b35-adc9-9c6174ba0b6e","Type":"ContainerDied","Data":"b2e4050779ab2931ab299f3b0d35f6771db4ad3083047a5be9c2d7ce582df077"} Sep 11 14:00:02.895690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:02.895657 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 14:00:02.935027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:02.934999 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 14:00:03.030269 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030200 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-util\") pod \"43a5de0d-249b-4e80-a955-01133ca9a3ca\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " Sep 11 14:00:03.030269 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030234 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jlv8m\" (UniqueName: \"kubernetes.io/projected/497fb332-ef1f-488e-8865-953fad0af090-kube-api-access-jlv8m\") pod \"497fb332-ef1f-488e-8865-953fad0af090\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " Sep 11 14:00:03.030449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030291 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-util\") pod \"497fb332-ef1f-488e-8865-953fad0af090\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " Sep 11 14:00:03.030449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030314 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-bundle\") pod \"497fb332-ef1f-488e-8865-953fad0af090\" (UID: \"497fb332-ef1f-488e-8865-953fad0af090\") " Sep 11 14:00:03.030449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030349 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lz5vg\" (UniqueName: \"kubernetes.io/projected/43a5de0d-249b-4e80-a955-01133ca9a3ca-kube-api-access-lz5vg\") pod \"43a5de0d-249b-4e80-a955-01133ca9a3ca\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " Sep 11 14:00:03.030449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030430 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-bundle\") pod \"43a5de0d-249b-4e80-a955-01133ca9a3ca\" (UID: \"43a5de0d-249b-4e80-a955-01133ca9a3ca\") " Sep 11 14:00:03.030944 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.030912 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-bundle" (OuterVolumeSpecName: "bundle") pod "497fb332-ef1f-488e-8865-953fad0af090" (UID: "497fb332-ef1f-488e-8865-953fad0af090"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:03.031248 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.031190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-bundle" (OuterVolumeSpecName: "bundle") pod "43a5de0d-249b-4e80-a955-01133ca9a3ca" (UID: "43a5de0d-249b-4e80-a955-01133ca9a3ca"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:03.032560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.032528 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43a5de0d-249b-4e80-a955-01133ca9a3ca-kube-api-access-lz5vg" (OuterVolumeSpecName: "kube-api-access-lz5vg") pod "43a5de0d-249b-4e80-a955-01133ca9a3ca" (UID: "43a5de0d-249b-4e80-a955-01133ca9a3ca"). InnerVolumeSpecName "kube-api-access-lz5vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:03.032642 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.032622 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497fb332-ef1f-488e-8865-953fad0af090-kube-api-access-jlv8m" (OuterVolumeSpecName: "kube-api-access-jlv8m") pod "497fb332-ef1f-488e-8865-953fad0af090" (UID: "497fb332-ef1f-488e-8865-953fad0af090"). InnerVolumeSpecName "kube-api-access-jlv8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:03.036405 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.036384 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-util" (OuterVolumeSpecName: "util") pod "43a5de0d-249b-4e80-a955-01133ca9a3ca" (UID: "43a5de0d-249b-4e80-a955-01133ca9a3ca"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:03.036480 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.036466 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-util" (OuterVolumeSpecName: "util") pod "497fb332-ef1f-488e-8865-953fad0af090" (UID: "497fb332-ef1f-488e-8865-953fad0af090"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:03.131876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.131850 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:03.131876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.131874 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlv8m\" (UniqueName: \"kubernetes.io/projected/497fb332-ef1f-488e-8865-953fad0af090-kube-api-access-jlv8m\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:03.132003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.131884 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:03.132003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.131893 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/497fb332-ef1f-488e-8865-953fad0af090-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:03.132003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.131902 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz5vg\" (UniqueName: \"kubernetes.io/projected/43a5de0d-249b-4e80-a955-01133ca9a3ca-kube-api-access-lz5vg\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:03.132003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.131909 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/43a5de0d-249b-4e80-a955-01133ca9a3ca-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:03.756358 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.756338 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" Sep 11 14:00:03.756692 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.756338 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c696w96" event={"ID":"43a5de0d-249b-4e80-a955-01133ca9a3ca","Type":"ContainerDied","Data":"50fe781d1f9c9d86bc43e02ead18963ec364f9c75fde222cf4a9211043d8e4d9"} Sep 11 14:00:03.756692 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.756486 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50fe781d1f9c9d86bc43e02ead18963ec364f9c75fde222cf4a9211043d8e4d9" Sep 11 14:00:03.758574 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.758550 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" event={"ID":"497fb332-ef1f-488e-8865-953fad0af090","Type":"ContainerDied","Data":"e0a3dd949005a602f065ffe12879b3a9a1c074fa76249f783e3979c742753ffe"} Sep 11 14:00:03.758668 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.758588 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0a3dd949005a602f065ffe12879b3a9a1c074fa76249f783e3979c742753ffe" Sep 11 14:00:03.758705 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.758678 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ctlrzd" Sep 11 14:00:03.760481 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.760455 2579 generic.go:356] "Generic (PLEG): container finished" podID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerID="74affc323b05f77ce2d3907a4db4479b350d950baa7c3403a9821c25e0c4df1a" exitCode=0 Sep 11 14:00:03.760593 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.760536 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" event={"ID":"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6","Type":"ContainerDied","Data":"74affc323b05f77ce2d3907a4db4479b350d950baa7c3403a9821c25e0c4df1a"} Sep 11 14:00:03.953883 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:03.953835 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 14:00:04.039122 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.039080 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-util\") pod \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " Sep 11 14:00:04.039250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.039186 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fpz2\" (UniqueName: \"kubernetes.io/projected/160460e1-29a3-4b35-adc9-9c6174ba0b6e-kube-api-access-4fpz2\") pod \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " Sep 11 14:00:04.039327 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.039250 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-bundle\") pod \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\" (UID: \"160460e1-29a3-4b35-adc9-9c6174ba0b6e\") " Sep 11 14:00:04.039796 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.039769 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-bundle" (OuterVolumeSpecName: "bundle") pod "160460e1-29a3-4b35-adc9-9c6174ba0b6e" (UID: "160460e1-29a3-4b35-adc9-9c6174ba0b6e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:04.041408 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.041385 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/160460e1-29a3-4b35-adc9-9c6174ba0b6e-kube-api-access-4fpz2" (OuterVolumeSpecName: "kube-api-access-4fpz2") pod "160460e1-29a3-4b35-adc9-9c6174ba0b6e" (UID: "160460e1-29a3-4b35-adc9-9c6174ba0b6e"). InnerVolumeSpecName "kube-api-access-4fpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:04.044321 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.044282 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-util" (OuterVolumeSpecName: "util") pod "160460e1-29a3-4b35-adc9-9c6174ba0b6e" (UID: "160460e1-29a3-4b35-adc9-9c6174ba0b6e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:04.140297 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.140270 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:04.140297 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.140295 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fpz2\" (UniqueName: \"kubernetes.io/projected/160460e1-29a3-4b35-adc9-9c6174ba0b6e-kube-api-access-4fpz2\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:04.140444 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.140305 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/160460e1-29a3-4b35-adc9-9c6174ba0b6e-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:04.770286 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.770257 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" Sep 11 14:00:04.770707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.770296 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a6k4br" event={"ID":"160460e1-29a3-4b35-adc9-9c6174ba0b6e","Type":"ContainerDied","Data":"c888e038456b4e1d1fd1d92935f71d0aec2e8f25f474cb3553080fd40a41ba2a"} Sep 11 14:00:04.770707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.770334 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c888e038456b4e1d1fd1d92935f71d0aec2e8f25f474cb3553080fd40a41ba2a" Sep 11 14:00:04.772440 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.772408 2579 generic.go:356] "Generic (PLEG): container finished" podID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerID="704114cb771d3d513d97cdcd4bc2ffc076d2e656ab3bed89f8eedd5b5668cfa5" exitCode=0 Sep 11 14:00:04.772586 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:04.772504 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" event={"ID":"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6","Type":"ContainerDied","Data":"704114cb771d3d513d97cdcd4bc2ffc076d2e656ab3bed89f8eedd5b5668cfa5"} Sep 11 14:00:05.914832 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:05.914809 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 14:00:06.055329 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.055310 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-bundle\") pod \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " Sep 11 14:00:06.055467 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.055342 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvsbd\" (UniqueName: \"kubernetes.io/projected/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-kube-api-access-tvsbd\") pod \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " Sep 11 14:00:06.055467 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.055374 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-util\") pod \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\" (UID: \"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6\") " Sep 11 14:00:06.055774 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.055751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-bundle" (OuterVolumeSpecName: "bundle") pod "a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" (UID: "a82aa63a-659a-41a0-acb3-1d10fbc3f5a6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:06.057727 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.057695 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-kube-api-access-tvsbd" (OuterVolumeSpecName: "kube-api-access-tvsbd") pod "a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" (UID: "a82aa63a-659a-41a0-acb3-1d10fbc3f5a6"). InnerVolumeSpecName "kube-api-access-tvsbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:06.060273 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.060249 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-util" (OuterVolumeSpecName: "util") pod "a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" (UID: "a82aa63a-659a-41a0-acb3-1d10fbc3f5a6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:06.156831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.156803 2579 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:06.156831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.156827 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvsbd\" (UniqueName: \"kubernetes.io/projected/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-kube-api-access-tvsbd\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:06.156831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.156837 2579 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a82aa63a-659a-41a0-acb3-1d10fbc3f5a6-util\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:06.784839 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.784809 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" Sep 11 14:00:06.785032 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.784842 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wpbvm" event={"ID":"a82aa63a-659a-41a0-acb3-1d10fbc3f5a6","Type":"ContainerDied","Data":"f87e45bf21fa2a3eae829ef31db00d02ab94d30b841b75086a784061a6533513"} Sep 11 14:00:06.785032 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:06.784883 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f87e45bf21fa2a3eae829ef31db00d02ab94d30b841b75086a784061a6533513" Sep 11 14:00:09.609505 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:09.609468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:11.654701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.654665 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz"] Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655001 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655011 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655018 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655024 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655032 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655037 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655043 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655048 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655058 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655064 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655070 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655075 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655083 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655087 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655094 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655111 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655121 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655126 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655133 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655139 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="extract" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655146 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655151 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="pull" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655156 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="util" Sep 11 14:00:11.655180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655161 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="util" Sep 11 14:00:11.656011 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655212 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="497fb332-ef1f-488e-8865-953fad0af090" containerName="extract" Sep 11 14:00:11.656011 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655220 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="160460e1-29a3-4b35-adc9-9c6174ba0b6e" containerName="extract" Sep 11 14:00:11.656011 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655228 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="43a5de0d-249b-4e80-a955-01133ca9a3ca" containerName="extract" Sep 11 14:00:11.656011 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.655234 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a82aa63a-659a-41a0-acb3-1d10fbc3f5a6" containerName="extract" Sep 11 14:00:11.658579 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.658560 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:11.661927 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.661907 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Sep 11 14:00:11.662068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.661906 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Sep 11 14:00:11.662676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.662658 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dfctm\"" Sep 11 14:00:11.674814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.674793 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz"] Sep 11 14:00:11.801834 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.801805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4l8t\" (UniqueName: \"kubernetes.io/projected/db81a821-075d-4a97-8750-174e27a9141e-kube-api-access-j4l8t\") pod \"limitador-operator-controller-manager-74495c69f8-tsjgz\" (UID: \"db81a821-075d-4a97-8750-174e27a9141e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:11.902542 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.902510 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4l8t\" (UniqueName: \"kubernetes.io/projected/db81a821-075d-4a97-8750-174e27a9141e-kube-api-access-j4l8t\") pod \"limitador-operator-controller-manager-74495c69f8-tsjgz\" (UID: \"db81a821-075d-4a97-8750-174e27a9141e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:11.916407 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.916346 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4l8t\" (UniqueName: \"kubernetes.io/projected/db81a821-075d-4a97-8750-174e27a9141e-kube-api-access-j4l8t\") pod \"limitador-operator-controller-manager-74495c69f8-tsjgz\" (UID: \"db81a821-075d-4a97-8750-174e27a9141e\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:11.968460 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:11.968435 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:12.251872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:12.251782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:12.254400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:12.254375 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz"] Sep 11 14:00:12.255818 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:00:12.255799 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb81a821_075d_4a97_8750_174e27a9141e.slice/crio-ab3c9c34bd94f9e3b546b7e88c311d2fcaeb44964d75dded484f13eff754bdab WatchSource:0}: Error finding container ab3c9c34bd94f9e3b546b7e88c311d2fcaeb44964d75dded484f13eff754bdab: Status 404 returned error can't find the container with id ab3c9c34bd94f9e3b546b7e88c311d2fcaeb44964d75dded484f13eff754bdab Sep 11 14:00:12.615146 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:12.615090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:12.818921 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:12.818890 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" event={"ID":"db81a821-075d-4a97-8750-174e27a9141e","Type":"ContainerStarted","Data":"ab3c9c34bd94f9e3b546b7e88c311d2fcaeb44964d75dded484f13eff754bdab"} Sep 11 14:00:13.005563 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:13.005461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:13.970774 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:13.970741 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9cf8c5469-wg6rb"] Sep 11 14:00:13.985937 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:13.983334 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:13.990308 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:13.990239 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cf8c5469-wg6rb"] Sep 11 14:00:14.055142 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.055088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:14.120244 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-trusted-ca-bundle\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.120458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120292 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-config\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.120458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120332 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-oauth-serving-cert\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.120458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120358 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-serving-cert\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.120458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120393 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd2r5\" (UniqueName: \"kubernetes.io/projected/a9a68ec6-3272-4926-97cb-8a59561dd60f-kube-api-access-sd2r5\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.120778 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-oauth-config\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.120778 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.120512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-service-ca\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221017 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.220938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-config\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221017 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.220979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-oauth-serving-cert\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221017 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-serving-cert\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd2r5\" (UniqueName: \"kubernetes.io/projected/a9a68ec6-3272-4926-97cb-8a59561dd60f-kube-api-access-sd2r5\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-oauth-config\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221586 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-service-ca\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221667 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-trusted-ca-bundle\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221667 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221658 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-config\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.221777 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.221756 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-oauth-serving-cert\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.222400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.222376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-service-ca\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.222628 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.222604 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a9a68ec6-3272-4926-97cb-8a59561dd60f-trusted-ca-bundle\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.224604 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.224579 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-serving-cert\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.224827 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.224809 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a9a68ec6-3272-4926-97cb-8a59561dd60f-console-oauth-config\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.230402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.230376 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd2r5\" (UniqueName: \"kubernetes.io/projected/a9a68ec6-3272-4926-97cb-8a59561dd60f-kube-api-access-sd2r5\") pod \"console-9cf8c5469-wg6rb\" (UID: \"a9a68ec6-3272-4926-97cb-8a59561dd60f\") " pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.297044 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.297010 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:14.474775 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:14.474680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:15.582301 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.582273 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9cf8c5469-wg6rb"] Sep 11 14:00:15.583093 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:00:15.583067 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a68ec6_3272_4926_97cb_8a59561dd60f.slice/crio-464c74729ba77c8374c01bf4c60ba041ef740e414a6b77433143dc40df8affe9 WatchSource:0}: Error finding container 464c74729ba77c8374c01bf4c60ba041ef740e414a6b77433143dc40df8affe9: Status 404 returned error can't find the container with id 464c74729ba77c8374c01bf4c60ba041ef740e414a6b77433143dc40df8affe9 Sep 11 14:00:15.664652 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.664618 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:15.846249 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.846166 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" event={"ID":"db81a821-075d-4a97-8750-174e27a9141e","Type":"ContainerStarted","Data":"dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743"} Sep 11 14:00:15.846412 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.846255 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:15.847941 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.847915 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cf8c5469-wg6rb" event={"ID":"a9a68ec6-3272-4926-97cb-8a59561dd60f","Type":"ContainerStarted","Data":"e2efc81b4c719d236659c314c34ff4b6119764effad1d0d978cd2e6d716042fe"} Sep 11 14:00:15.848067 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.847955 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9cf8c5469-wg6rb" event={"ID":"a9a68ec6-3272-4926-97cb-8a59561dd60f","Type":"ContainerStarted","Data":"464c74729ba77c8374c01bf4c60ba041ef740e414a6b77433143dc40df8affe9"} Sep 11 14:00:15.876929 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.876883 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" podStartSLOduration=1.793500683 podStartE2EDuration="4.876867804s" totalImagesPullingTime="3.083367121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:00:15.875485284 +0000 UTC m=+432.745216547" watchObservedRunningTime="2026-09-11 14:00:15.876867804 +0000 UTC m=+432.746599067" Sep 11 14:00:15.914670 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:15.914623 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9cf8c5469-wg6rb" podStartSLOduration=2.9146084869999997 podStartE2EDuration="2.914608487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:00:15.910894363 +0000 UTC m=+432.780625637" watchObservedRunningTime="2026-09-11 14:00:15.914608487 +0000 UTC m=+432.784339748" Sep 11 14:00:16.596407 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:16.596373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:16.722793 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:16.722759 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:18.448213 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:18.448182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:18.598478 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:18.598451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:18.931939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:18.931901 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:20.711568 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.711539 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2cnmk"] Sep 11 14:00:20.715235 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.715218 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:20.717763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.717745 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-8gfzl\"" Sep 11 14:00:20.724990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.724964 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2cnmk"] Sep 11 14:00:20.878641 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.878612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4jhp\" (UniqueName: \"kubernetes.io/projected/59493059-1289-4d1c-99b9-d4b3d1603d61-kube-api-access-s4jhp\") pod \"authorino-operator-76b478bd78-2cnmk\" (UID: \"59493059-1289-4d1c-99b9-d4b3d1603d61\") " pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:20.979360 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.979281 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4jhp\" (UniqueName: \"kubernetes.io/projected/59493059-1289-4d1c-99b9-d4b3d1603d61-kube-api-access-s4jhp\") pod \"authorino-operator-76b478bd78-2cnmk\" (UID: \"59493059-1289-4d1c-99b9-d4b3d1603d61\") " pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:20.994449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:20.994420 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4jhp\" (UniqueName: \"kubernetes.io/projected/59493059-1289-4d1c-99b9-d4b3d1603d61-kube-api-access-s4jhp\") pod \"authorino-operator-76b478bd78-2cnmk\" (UID: \"59493059-1289-4d1c-99b9-d4b3d1603d61\") " pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:21.025359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:21.025336 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:21.341760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:21.341734 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-2cnmk"] Sep 11 14:00:21.342699 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:00:21.342669 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59493059_1289_4d1c_99b9_d4b3d1603d61.slice/crio-7ce692ebdd0dd65353aab723b05e31560ceae25d69a58e07d511106da9b108bb WatchSource:0}: Error finding container 7ce692ebdd0dd65353aab723b05e31560ceae25d69a58e07d511106da9b108bb: Status 404 returned error can't find the container with id 7ce692ebdd0dd65353aab723b05e31560ceae25d69a58e07d511106da9b108bb Sep 11 14:00:21.582130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:21.582071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:21.887412 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:21.887380 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" event={"ID":"59493059-1289-4d1c-99b9-d4b3d1603d61","Type":"ContainerStarted","Data":"7ce692ebdd0dd65353aab723b05e31560ceae25d69a58e07d511106da9b108bb"} Sep 11 14:00:22.268336 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:22.268249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:22.387705 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:22.387664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:23.400951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:23.400918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:23.945793 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:23.945760 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:24.298044 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:24.298009 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:24.298254 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:24.298074 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:24.302975 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:24.302957 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:24.909901 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:24.909876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9cf8c5469-wg6rb" Sep 11 14:00:24.965480 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:24.965452 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b9dc8fd87-vhjcc"] Sep 11 14:00:25.067929 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:25.067896 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:25.168774 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:25.168684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:25.252344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:25.252293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:26.225280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.225189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:26.331038 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.331006 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:26.620927 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.620889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:26.721448 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.721416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:26.857668 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.857646 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:26.919772 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.919692 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" event={"ID":"59493059-1289-4d1c-99b9-d4b3d1603d61","Type":"ContainerStarted","Data":"a244f07e8545b9c9db8455b51dbd510b7a6a8d4da856181d527f50b7136455da"} Sep 11 14:00:26.919917 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.919830 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:26.938724 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:26.938678 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" podStartSLOduration=1.921963242 podStartE2EDuration="6.938664996s" totalImagesPullingTime="5.016701754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:00:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:00:26.936853406 +0000 UTC m=+443.806584692" watchObservedRunningTime="2026-09-11 14:00:26.938664996 +0000 UTC m=+443.808396258" Sep 11 14:00:27.011230 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:27.011200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:27.114163 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:27.114129 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:30.274124 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:30.274075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:30.381969 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:30.381939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:32.698494 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:32.698459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:32.844128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:32.844072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:34.134063 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.134029 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc"] Sep 11 14:00:34.137954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.137932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.140814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.140789 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qc8w5\"" Sep 11 14:00:34.151915 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.151892 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc"] Sep 11 14:00:34.188865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.188837 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbc392d7-df81-4b4d-a0ed-af2949e68204-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.188980 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.188874 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vd942\" (UniqueName: \"kubernetes.io/projected/dbc392d7-df81-4b4d-a0ed-af2949e68204-kube-api-access-vd942\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.290276 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.290244 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbc392d7-df81-4b4d-a0ed-af2949e68204-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.290400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.290294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vd942\" (UniqueName: \"kubernetes.io/projected/dbc392d7-df81-4b4d-a0ed-af2949e68204-kube-api-access-vd942\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.290606 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.290583 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbc392d7-df81-4b4d-a0ed-af2949e68204-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.299774 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.299753 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vd942\" (UniqueName: \"kubernetes.io/projected/dbc392d7-df81-4b4d-a0ed-af2949e68204-kube-api-access-vd942\") pod \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.447896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.447834 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:34.477542 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.477510 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc"] Sep 11 14:00:34.498445 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.498410 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc"] Sep 11 14:00:34.507461 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.507389 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz"] Sep 11 14:00:34.507650 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.507624 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" podUID="db81a821-075d-4a97-8750-174e27a9141e" containerName="manager" containerID="cri-o://dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743" gracePeriod=2 Sep 11 14:00:34.529861 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.529836 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz"] Sep 11 14:00:34.540540 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:34.540498 2579 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"db81a821-075d-4a97-8750-174e27a9141e\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-74495c69f8-tsjgz\": pods \"limitador-operator-controller-manager-74495c69f8-tsjgz\" not found" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:34.558490 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.558462 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59"] Sep 11 14:00:34.558834 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.558822 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db81a821-075d-4a97-8750-174e27a9141e" containerName="manager" Sep 11 14:00:34.558897 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.558836 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db81a821-075d-4a97-8750-174e27a9141e" containerName="manager" Sep 11 14:00:34.558951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.558900 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db81a821-075d-4a97-8750-174e27a9141e" containerName="manager" Sep 11 14:00:34.561961 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.561945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:34.564561 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:34.564536 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tsjgz\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="db81a821-075d-4a97-8750-174e27a9141e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:34.584064 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.584037 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59"] Sep 11 14:00:34.592880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.592858 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxg7h\" (UniqueName: \"kubernetes.io/projected/8ce4f934-4c13-4fef-a4af-4136e1038b1b-kube-api-access-nxg7h\") pod \"limitador-operator-controller-manager-74495c69f8-99d59\" (UID: \"8ce4f934-4c13-4fef-a4af-4136e1038b1b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:34.693738 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.693710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxg7h\" (UniqueName: \"kubernetes.io/projected/8ce4f934-4c13-4fef-a4af-4136e1038b1b-kube-api-access-nxg7h\") pod \"limitador-operator-controller-manager-74495c69f8-99d59\" (UID: \"8ce4f934-4c13-4fef-a4af-4136e1038b1b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:34.704559 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.704495 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxg7h\" (UniqueName: \"kubernetes.io/projected/8ce4f934-4c13-4fef-a4af-4136e1038b1b-kube-api-access-nxg7h\") pod \"limitador-operator-controller-manager-74495c69f8-99d59\" (UID: \"8ce4f934-4c13-4fef-a4af-4136e1038b1b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:34.741227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.741203 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:34.743305 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:34.743281 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tsjgz\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="db81a821-075d-4a97-8750-174e27a9141e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:34.794733 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.794710 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4l8t\" (UniqueName: \"kubernetes.io/projected/db81a821-075d-4a97-8750-174e27a9141e-kube-api-access-j4l8t\") pod \"db81a821-075d-4a97-8750-174e27a9141e\" (UID: \"db81a821-075d-4a97-8750-174e27a9141e\") " Sep 11 14:00:34.796881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.796859 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db81a821-075d-4a97-8750-174e27a9141e-kube-api-access-j4l8t" (OuterVolumeSpecName: "kube-api-access-j4l8t") pod "db81a821-075d-4a97-8750-174e27a9141e" (UID: "db81a821-075d-4a97-8750-174e27a9141e"). InnerVolumeSpecName "kube-api-access-j4l8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:34.893318 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.893293 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:34.895655 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.895636 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4l8t\" (UniqueName: \"kubernetes.io/projected/db81a821-075d-4a97-8750-174e27a9141e-kube-api-access-j4l8t\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:34.966462 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.966379 2579 generic.go:356] "Generic (PLEG): container finished" podID="db81a821-075d-4a97-8750-174e27a9141e" containerID="dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743" exitCode=0 Sep 11 14:00:34.966462 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.966449 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:34.966850 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.966455 2579 scope.go:117] "RemoveContainer" containerID="dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743" Sep 11 14:00:34.969023 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:34.968997 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tsjgz\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="db81a821-075d-4a97-8750-174e27a9141e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:34.976758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.976740 2579 scope.go:117] "RemoveContainer" containerID="dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743" Sep 11 14:00:34.977030 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:34.977011 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743\": container with ID starting with dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743 not found: ID does not exist" containerID="dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743" Sep 11 14:00:34.977083 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:34.977039 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743"} err="failed to get container status \"dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743\": rpc error: code = NotFound desc = could not find container \"dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743\": container with ID starting with dd74e4b611aa9eef445d3955decb4ec0cca1b246099c638ae3b588e6655c0743 not found: ID does not exist" Sep 11 14:00:34.977721 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:34.977696 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-tsjgz\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="db81a821-075d-4a97-8750-174e27a9141e" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-tsjgz" Sep 11 14:00:35.171240 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.171046 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59"] Sep 11 14:00:35.174245 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:00:35.174221 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce4f934_4c13_4fef_a4af_4136e1038b1b.slice/crio-e7d51fdf23de1777d7936c5e6e136ac26aef4b13c74c09e71be37cd8f5de5fa7 WatchSource:0}: Error finding container e7d51fdf23de1777d7936c5e6e136ac26aef4b13c74c09e71be37cd8f5de5fa7: Status 404 returned error can't find the container with id e7d51fdf23de1777d7936c5e6e136ac26aef4b13c74c09e71be37cd8f5de5fa7 Sep 11 14:00:35.431066 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:35.431032 2579 log.go:32] "RunPodSandbox from runtime service failed" err=< Sep 11 14:00:35.431066 ip-10-0-136-162 kubenswrapper[2579]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-nf9rc_kuadrant-system_dbc392d7-df81-4b4d-a0ed-af2949e68204_0(88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-nf9rc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610" Netns:"/var/run/netns/0f0a544f-1806-42ad-b135-2ead111d4a9e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-nf9rc;K8S_POD_INFRA_CONTAINER_ID=88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610;K8S_POD_UID=dbc392d7-df81-4b4d-a0ed-af2949e68204" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc/dbc392d7-df81-4b4d-a0ed-af2949e68204:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc 88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610 network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" not found Sep 11 14:00:35.431066 ip-10-0-136-162 kubenswrapper[2579]: ' Sep 11 14:00:35.431066 ip-10-0-136-162 kubenswrapper[2579]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 14:00:35.431066 ip-10-0-136-162 kubenswrapper[2579]: > Sep 11 14:00:35.431389 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:35.431093 2579 kuberuntime_sandbox.go:71] "Failed to create sandbox for pod" err=< Sep 11 14:00:35.431389 ip-10-0-136-162 kubenswrapper[2579]: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_kuadrant-operator-controller-manager-7978cbdfbc-nf9rc_kuadrant-system_dbc392d7-df81-4b4d-a0ed-af2949e68204_0(88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610): error adding pod kuadrant-system_kuadrant-operator-controller-manager-7978cbdfbc-nf9rc to CNI network "multus-cni-network": plugin type="multus-shim" name="multus-cni-network" failed (add): CmdAdd (shim): CNI request failed with status 400: 'ContainerID:"88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610" Netns:"/var/run/netns/0f0a544f-1806-42ad-b135-2ead111d4a9e" IfName:"eth0" Args:"IgnoreUnknown=1;K8S_POD_NAMESPACE=kuadrant-system;K8S_POD_NAME=kuadrant-operator-controller-manager-7978cbdfbc-nf9rc;K8S_POD_INFRA_CONTAINER_ID=88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610;K8S_POD_UID=dbc392d7-df81-4b4d-a0ed-af2949e68204" Path:"" ERRORED: error configuring pod [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc] networking: [kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc/dbc392d7-df81-4b4d-a0ed-af2949e68204:ovn-kubernetes]: error adding container to network "ovn-kubernetes": CNI request failed with status 400: '[kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc 88dc43b038ad655281feb36a01b39450c6c1807507475e93147b5f6268b95610 network default NAD default NAD key default] failed to get pod kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc: timed out waiting for pod after 1s: pods "kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" not found Sep 11 14:00:35.431389 ip-10-0-136-162 kubenswrapper[2579]: ' Sep 11 14:00:35.431389 ip-10-0-136-162 kubenswrapper[2579]: ': StdinData: {"auxiliaryCNIChainName":"vendor-cni-chain","binDir":"/var/lib/cni/bin","clusterNetwork":"/host/run/multus/cni/net.d/10-ovn-kubernetes.conf","cniVersion":"0.3.1","daemonSocketDir":"/run/multus/socket","globalNamespaces":"default,openshift-multus,openshift-sriov-network-operator,openshift-cnv","logLevel":"verbose","logToStderr":true,"name":"multus-cni-network","namespaceIsolation":true,"type":"multus-shim"} Sep 11 14:00:35.431389 ip-10-0-136-162 kubenswrapper[2579]: > pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:35.643319 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.643288 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db81a821-075d-4a97-8750-174e27a9141e" path="/var/lib/kubelet/pods/db81a821-075d-4a97-8750-174e27a9141e/volumes" Sep 11 14:00:35.973704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.973615 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" event={"ID":"8ce4f934-4c13-4fef-a4af-4136e1038b1b","Type":"ContainerStarted","Data":"b4fb66dbf25a6d16bc14a64ae0d745c3b21fd7e930cf4b24e668773b5c6799c0"} Sep 11 14:00:35.973704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.973665 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" event={"ID":"8ce4f934-4c13-4fef-a4af-4136e1038b1b","Type":"ContainerStarted","Data":"e7d51fdf23de1777d7936c5e6e136ac26aef4b13c74c09e71be37cd8f5de5fa7"} Sep 11 14:00:35.973704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.973703 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:35.974685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.974667 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:35.982815 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.982796 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:35.994914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:35.994867 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" podStartSLOduration=1.994850996 podStartE2EDuration="1.994850996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:00:35.994153759 +0000 UTC m=+452.863885044" watchObservedRunningTime="2026-09-11 14:00:35.994850996 +0000 UTC m=+452.864582260" Sep 11 14:00:35.996515 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:35.996493 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="dbc392d7-df81-4b4d-a0ed-af2949e68204" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:36.004785 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.004768 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vd942\" (UniqueName: \"kubernetes.io/projected/dbc392d7-df81-4b4d-a0ed-af2949e68204-kube-api-access-vd942\") pod \"dbc392d7-df81-4b4d-a0ed-af2949e68204\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " Sep 11 14:00:36.004855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.004798 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbc392d7-df81-4b4d-a0ed-af2949e68204-extensions-socket-volume\") pod \"dbc392d7-df81-4b4d-a0ed-af2949e68204\" (UID: \"dbc392d7-df81-4b4d-a0ed-af2949e68204\") " Sep 11 14:00:36.005078 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.005057 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbc392d7-df81-4b4d-a0ed-af2949e68204-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "dbc392d7-df81-4b4d-a0ed-af2949e68204" (UID: "dbc392d7-df81-4b4d-a0ed-af2949e68204"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:00:36.006918 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.006895 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc392d7-df81-4b4d-a0ed-af2949e68204-kube-api-access-vd942" (OuterVolumeSpecName: "kube-api-access-vd942") pod "dbc392d7-df81-4b4d-a0ed-af2949e68204" (UID: "dbc392d7-df81-4b4d-a0ed-af2949e68204"). InnerVolumeSpecName "kube-api-access-vd942". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:36.106277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.106245 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vd942\" (UniqueName: \"kubernetes.io/projected/dbc392d7-df81-4b4d-a0ed-af2949e68204-kube-api-access-vd942\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:36.106277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.106272 2579 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dbc392d7-df81-4b4d-a0ed-af2949e68204-extensions-socket-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:36.979690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:36.979650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:36.982146 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:36.982116 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="dbc392d7-df81-4b4d-a0ed-af2949e68204" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:36.987344 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:36.987318 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-7978cbdfbc-nf9rc\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="dbc392d7-df81-4b4d-a0ed-af2949e68204" pod="kuadrant-system/kuadrant-operator-controller-manager-7978cbdfbc-nf9rc" Sep 11 14:00:37.643053 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:37.643020 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbc392d7-df81-4b4d-a0ed-af2949e68204" path="/var/lib/kubelet/pods/dbc392d7-df81-4b4d-a0ed-af2949e68204/volumes" Sep 11 14:00:37.928168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:37.928082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-2cnmk" Sep 11 14:00:39.268257 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:39.268225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:39.367598 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:39.367566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:41.848747 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:41.848713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:41.950641 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:41.950603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:45.107793 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:45.107766 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:45.226492 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:45.226459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:45.343856 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:45.343827 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:46.982195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:46.982165 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-99d59" Sep 11 14:00:49.985474 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:49.985423 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7b9dc8fd87-vhjcc" podUID="bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" containerName="console" containerID="cri-o://a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e" gracePeriod=15 Sep 11 14:00:50.226414 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.226391 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 14:00:50.322903 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.322871 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-trusted-ca-bundle\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.322929 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xfzgt\" (UniqueName: \"kubernetes.io/projected/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-kube-api-access-xfzgt\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.322963 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-config\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.322990 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-oauth-config\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323021 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-oauth-serving-cert\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323056 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-service-ca\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323363 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323081 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-serving-cert\") pod \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\" (UID: \"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07\") " Sep 11 14:00:50.323425 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:00:50.323496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323435 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:00:50.323496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323441 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-config" (OuterVolumeSpecName: "console-config") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:00:50.323614 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.323516 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-service-ca" (OuterVolumeSpecName: "service-ca") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:00:50.325465 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.325440 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:00:50.325562 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.325511 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:00:50.325562 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.325549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-kube-api-access-xfzgt" (OuterVolumeSpecName: "kube-api-access-xfzgt") pod "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" (UID: "bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07"). InnerVolumeSpecName "kube-api-access-xfzgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:00:50.424252 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424220 2579 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-trusted-ca-bundle\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:50.424252 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424244 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfzgt\" (UniqueName: \"kubernetes.io/projected/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-kube-api-access-xfzgt\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:50.424252 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424255 2579 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-config\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:50.424510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424264 2579 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-oauth-config\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:50.424510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424275 2579 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-oauth-serving-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:50.424510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424283 2579 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:50.424510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:50.424291 2579 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07-console-serving-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:00:51.059950 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.059918 2579 generic.go:356] "Generic (PLEG): container finished" podID="bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" containerID="a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e" exitCode=0 Sep 11 14:00:51.060362 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.059963 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7b9dc8fd87-vhjcc" Sep 11 14:00:51.060362 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.060005 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9dc8fd87-vhjcc" event={"ID":"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07","Type":"ContainerDied","Data":"a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e"} Sep 11 14:00:51.060362 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.060058 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7b9dc8fd87-vhjcc" event={"ID":"bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07","Type":"ContainerDied","Data":"eb012d57ee88b948ee6bbbe0094a904637670f77ec0c92020d67fdf385fdf0e2"} Sep 11 14:00:51.060362 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.060081 2579 scope.go:117] "RemoveContainer" containerID="a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e" Sep 11 14:00:51.069770 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.069754 2579 scope.go:117] "RemoveContainer" containerID="a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e" Sep 11 14:00:51.070072 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:00:51.070054 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e\": container with ID starting with a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e not found: ID does not exist" containerID="a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e" Sep 11 14:00:51.070145 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.070080 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e"} err="failed to get container status \"a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e\": rpc error: code = NotFound desc = could not find container \"a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e\": container with ID starting with a185ea0dad09aca20a1921c1af3bc01cfb238b975a5248b52fb7ee2f35bc925e not found: ID does not exist" Sep 11 14:00:51.086930 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.086906 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7b9dc8fd87-vhjcc"] Sep 11 14:00:51.092301 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.092281 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7b9dc8fd87-vhjcc"] Sep 11 14:00:51.643293 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:51.643263 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" path="/var/lib/kubelet/pods/bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07/volumes" Sep 11 14:00:53.735487 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:53.735455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:53.836729 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:53.836701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:59.172596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:59.172556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:00:59.270007 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:00:59.269956 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:01.780468 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:01.780410 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:01.881138 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:01.881071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:02.764590 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.764551 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j"] Sep 11 14:01:02.765220 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.765202 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" containerName="console" Sep 11 14:01:02.765281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.765224 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" containerName="console" Sep 11 14:01:02.765346 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.765336 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdc1ccd0-a84a-4930-9bb4-48d2c0a17f07" containerName="console" Sep 11 14:01:02.767950 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.767931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.770415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.770396 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5bmnj\"" Sep 11 14:01:02.780497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.780474 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j"] Sep 11 14:01:02.786976 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.786954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:02.830457 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830422 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830457 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830457 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830597 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830626 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v924m\" (UniqueName: \"kubernetes.io/projected/d91cba3f-bcf1-42cc-921f-f41d9be3f014-kube-api-access-v924m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830893 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830700 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.830893 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.830771 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.886844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.886810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:02.932087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932057 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932215 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v924m\" (UniqueName: \"kubernetes.io/projected/d91cba3f-bcf1-42cc-921f-f41d9be3f014-kube-api-access-v924m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932295 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932615 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932665 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.932936 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.932915 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.933228 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.933207 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.934842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.934824 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.935048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.935030 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.940168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.940147 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d91cba3f-bcf1-42cc-921f-f41d9be3f014-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:02.940483 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:02.940466 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v924m\" (UniqueName: \"kubernetes.io/projected/d91cba3f-bcf1-42cc-921f-f41d9be3f014-kube-api-access-v924m\") pod \"maas-default-gateway-openshift-default-78bb78ff78-vlz4j\" (UID: \"d91cba3f-bcf1-42cc-921f-f41d9be3f014\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:03.079170 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:03.079132 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:03.360848 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:03.360817 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j"] Sep 11 14:01:03.361507 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:01:03.361484 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd91cba3f_bcf1_42cc_921f_f41d9be3f014.slice/crio-371abd60f9d11e41bc5f55f11514d9e5b5bb579fe6ee10fbaef94c51155bf056 WatchSource:0}: Error finding container 371abd60f9d11e41bc5f55f11514d9e5b5bb579fe6ee10fbaef94c51155bf056: Status 404 returned error can't find the container with id 371abd60f9d11e41bc5f55f11514d9e5b5bb579fe6ee10fbaef94c51155bf056 Sep 11 14:01:03.363727 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:03.363699 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:01:03.363802 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:03.363762 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:01:03.363802 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:03.363787 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:01:04.137436 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:04.137405 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" event={"ID":"d91cba3f-bcf1-42cc-921f-f41d9be3f014","Type":"ContainerStarted","Data":"b847d68b0d0d20c7624b28b5155ffe865cf72527c34e833be9bfb8c84ed83b3d"} Sep 11 14:01:04.137810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:04.137456 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" event={"ID":"d91cba3f-bcf1-42cc-921f-f41d9be3f014","Type":"ContainerStarted","Data":"371abd60f9d11e41bc5f55f11514d9e5b5bb579fe6ee10fbaef94c51155bf056"} Sep 11 14:01:04.164989 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:04.164942 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" podStartSLOduration=2.164927335 podStartE2EDuration="2.164927335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:01:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:01:04.161997386 +0000 UTC m=+481.031728671" watchObservedRunningTime="2026-09-11 14:01:04.164927335 +0000 UTC m=+481.034658596" Sep 11 14:01:05.079823 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.079793 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:05.084624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.084602 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:05.143168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.143140 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:05.144039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.144017 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:01:05.570686 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.570659 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:05.671482 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.671453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:05.785858 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.785824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:05.892359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:05.892280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:06.776078 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:06.776044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:06.907881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:06.907845 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:13.607645 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:13.607611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:13.707619 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:13.707589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:17.415588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:17.415547 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:17.532040 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:17.532007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:17.610550 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:17.610514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:18.350805 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:18.350770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:18.454118 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:18.454067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:20.175965 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:20.175939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:20.291496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:20.291465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:36.686898 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:36.686856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:36.782641 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:36.782607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:37.185556 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:37.185525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:49.984826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:49.984787 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:50.340874 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:50.340832 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:57.065419 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:57.065377 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:57.236314 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:57.236277 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:57.390885 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:57.390769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:57.575604 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:57.575568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:01:57.763310 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:01:57.763221 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:00.587144 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:00.587112 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:00.737191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:00.737160 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:33.283202 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:33.283168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:33.436075 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:33.436039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:34.701310 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:34.701265 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:34.801814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:34.801773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:35.066388 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.066349 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:35.164969 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.164936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:35.268672 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.268636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:35.369535 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.369438 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:35.456673 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.456639 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5"] Sep 11 14:02:35.459250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.459228 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.465700 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.465676 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Sep 11 14:02:35.473812 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.473785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:35.476433 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.476410 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5"] Sep 11 14:02:35.550042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.550005 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f0606d-a1dc-43d2-9264-7e2c583ce576-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fzzg5\" (UID: \"06f0606d-a1dc-43d2-9264-7e2c583ce576\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.550237 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.550092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98gws\" (UniqueName: \"kubernetes.io/projected/06f0606d-a1dc-43d2-9264-7e2c583ce576-kube-api-access-98gws\") pod \"payload-pre-processing-59fbc47bf8-fzzg5\" (UID: \"06f0606d-a1dc-43d2-9264-7e2c583ce576\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.650730 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.650650 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98gws\" (UniqueName: \"kubernetes.io/projected/06f0606d-a1dc-43d2-9264-7e2c583ce576-kube-api-access-98gws\") pod \"payload-pre-processing-59fbc47bf8-fzzg5\" (UID: \"06f0606d-a1dc-43d2-9264-7e2c583ce576\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.650730 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.650719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f0606d-a1dc-43d2-9264-7e2c583ce576-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fzzg5\" (UID: \"06f0606d-a1dc-43d2-9264-7e2c583ce576\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.651427 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.651408 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f0606d-a1dc-43d2-9264-7e2c583ce576-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-fzzg5\" (UID: \"06f0606d-a1dc-43d2-9264-7e2c583ce576\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.666631 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.666600 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98gws\" (UniqueName: \"kubernetes.io/projected/06f0606d-a1dc-43d2-9264-7e2c583ce576-kube-api-access-98gws\") pod \"payload-pre-processing-59fbc47bf8-fzzg5\" (UID: \"06f0606d-a1dc-43d2-9264-7e2c583ce576\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:35.770379 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:35.770339 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:36.055878 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:36.055688 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5"] Sep 11 14:02:36.059616 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:02:36.059589 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f0606d_a1dc_43d2_9264_7e2c583ce576.slice/crio-40e491d5cf068d2e3051e17aa37365b08575e161e4ae694ba35bf7fd9f1cabdc WatchSource:0}: Error finding container 40e491d5cf068d2e3051e17aa37365b08575e161e4ae694ba35bf7fd9f1cabdc: Status 404 returned error can't find the container with id 40e491d5cf068d2e3051e17aa37365b08575e161e4ae694ba35bf7fd9f1cabdc Sep 11 14:02:36.653992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:36.653960 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" event={"ID":"06f0606d-a1dc-43d2-9264-7e2c583ce576","Type":"ContainerStarted","Data":"40e491d5cf068d2e3051e17aa37365b08575e161e4ae694ba35bf7fd9f1cabdc"} Sep 11 14:02:38.667626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:38.667579 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" event={"ID":"06f0606d-a1dc-43d2-9264-7e2c583ce576","Type":"ContainerStarted","Data":"d12280f246e7792208fac7efc7215d7213fbfdbbcc804df4e092276b0f703a75"} Sep 11 14:02:38.668067 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:38.667761 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:38.694197 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:38.693946 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" podStartSLOduration=1.593244238 podStartE2EDuration="3.693924735s" totalImagesPullingTime="2.100680497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:38.691612095 +0000 UTC m=+575.561343357" watchObservedRunningTime="2026-09-11 14:02:38.693924735 +0000 UTC m=+575.563655988" Sep 11 14:02:44.712565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:44.712534 2579 generic.go:356] "Generic (PLEG): container finished" podID="d91cba3f-bcf1-42cc-921f-f41d9be3f014" containerID="b847d68b0d0d20c7624b28b5155ffe865cf72527c34e833be9bfb8c84ed83b3d" exitCode=0 Sep 11 14:02:44.713020 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:44.712597 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" event={"ID":"d91cba3f-bcf1-42cc-921f-f41d9be3f014","Type":"ContainerDied","Data":"b847d68b0d0d20c7624b28b5155ffe865cf72527c34e833be9bfb8c84ed83b3d"} Sep 11 14:02:44.713020 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:44.712997 2579 scope.go:117] "RemoveContainer" containerID="b847d68b0d0d20c7624b28b5155ffe865cf72527c34e833be9bfb8c84ed83b3d" Sep 11 14:02:44.714027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:44.713997 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:02:44.714113 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:44.714082 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:02:44.714184 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:44.714171 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:02:45.071757 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:45.071712 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:45.166522 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:45.166482 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:45.720449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:45.720413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" event={"ID":"d91cba3f-bcf1-42cc-921f-f41d9be3f014","Type":"ContainerStarted","Data":"bdf729f28bd0b9ff349fc93950cd86bd81432d2ba066f4ac3adaff9da8736392"} Sep 11 14:02:46.079340 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:46.079290 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:02:46.080694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:46.080665 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vlz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:02:46.080941 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:46.080916 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" podUID="d91cba3f-bcf1-42cc-921f-f41d9be3f014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 11 14:02:47.080561 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:47.080511 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vlz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:02:47.081050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:47.080602 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" podUID="d91cba3f-bcf1-42cc-921f-f41d9be3f014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 11 14:02:48.079587 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:48.079548 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vlz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:02:48.079915 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:48.079885 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" podUID="d91cba3f-bcf1-42cc-921f-f41d9be3f014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 11 14:02:49.080318 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:49.080280 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vlz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:02:49.080944 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:49.080911 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" podUID="d91cba3f-bcf1-42cc-921f-f41d9be3f014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 11 14:02:49.679330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:49.679286 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-fzzg5" Sep 11 14:02:50.080087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:50.080037 2579 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-vlz4j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Sep 11 14:02:50.080322 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:50.080140 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" podUID="d91cba3f-bcf1-42cc-921f-f41d9be3f014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Sep 11 14:02:50.143875 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:50.143841 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:02:51.083951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:51.083924 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:02:51.085039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:51.085019 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-vlz4j" Sep 11 14:02:56.395676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.395639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:02:56.402928 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.402898 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl"] Sep 11 14:02:56.405375 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.405358 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.410234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.410212 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4r8ml\"" Sep 11 14:02:56.410770 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.410752 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Sep 11 14:02:56.411026 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.411001 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Sep 11 14:02:56.411812 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.411798 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Sep 11 14:02:56.440091 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.440065 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl"] Sep 11 14:02:56.477417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.477383 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.477417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.477417 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.477624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.477448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.477624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.477554 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssrk\" (UniqueName: \"kubernetes.io/projected/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-kube-api-access-nssrk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.477624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.477615 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.477727 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.477646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.578837 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.578787 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.578837 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.578839 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579066 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.578866 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579066 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.578883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579066 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.578912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579066 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.578955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssrk\" (UniqueName: \"kubernetes.io/projected/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-kube-api-access-nssrk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579325 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.579237 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579325 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.579313 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.579432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.579367 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.581374 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.581356 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.581614 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.581597 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.595896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.595865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssrk\" (UniqueName: \"kubernetes.io/projected/7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e-kube-api-access-nssrk\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl\" (UID: \"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.716619 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.716526 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:02:56.993766 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:56.993736 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl"] Sep 11 14:02:56.995115 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:02:56.995076 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1fe7ec_06f7_47cc_90fa_b233c76d5f3e.slice/crio-01a0f542c19997983537e10cd9112ea468ca6ebf90d155efd0bb4fb6a87c946d WatchSource:0}: Error finding container 01a0f542c19997983537e10cd9112ea468ca6ebf90d155efd0bb4fb6a87c946d: Status 404 returned error can't find the container with id 01a0f542c19997983537e10cd9112ea468ca6ebf90d155efd0bb4fb6a87c946d Sep 11 14:02:57.791315 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:57.791274 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" event={"ID":"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e","Type":"ContainerStarted","Data":"01a0f542c19997983537e10cd9112ea468ca6ebf90d155efd0bb4fb6a87c946d"} Sep 11 14:02:58.798971 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:58.798934 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" event={"ID":"7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e","Type":"ContainerStarted","Data":"39f193c12375dd72e1f167c2711621ff9ab835c63206a52c503f8437cafab0e7"} Sep 11 14:02:58.823527 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:02:58.823478 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" podStartSLOduration=1.703876336 podStartE2EDuration="2.823463222s" totalImagesPullingTime="1.119586886s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:02:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:02:58.82093965 +0000 UTC m=+595.690670914" watchObservedRunningTime="2026-09-11 14:02:58.823463222 +0000 UTC m=+595.693194502" Sep 11 14:03:06.716952 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:06.716919 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:03:06.717477 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:06.717221 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:03:06.730097 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:06.730071 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:03:06.854235 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:06.854205 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl" Sep 11 14:03:43.638418 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:43.638350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:55.584243 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.584204 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp"] Sep 11 14:03:55.587233 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.587203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.589939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.589919 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Sep 11 14:03:55.599514 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.599490 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp"] Sep 11 14:03:55.686174 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.686130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:03:55.741206 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.741166 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d334514-92f2-4d83-8791-4892bf569912-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.741383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.741214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.741383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.741240 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.741383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.741367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl5sg\" (UniqueName: \"kubernetes.io/projected/6d334514-92f2-4d83-8791-4892bf569912-kube-api-access-jl5sg\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.741514 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.741412 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.741695 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.741676 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.842922 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.842826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl5sg\" (UniqueName: \"kubernetes.io/projected/6d334514-92f2-4d83-8791-4892bf569912-kube-api-access-jl5sg\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.842922 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.842872 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.842932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.842959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d334514-92f2-4d83-8791-4892bf569912-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.842992 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.843008 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843424 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.843400 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843424 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.843411 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.843503 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.843436 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.845553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.845533 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d334514-92f2-4d83-8791-4892bf569912-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.845810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.845790 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d334514-92f2-4d83-8791-4892bf569912-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.852116 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.852076 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl5sg\" (UniqueName: \"kubernetes.io/projected/6d334514-92f2-4d83-8791-4892bf569912-kube-api-access-jl5sg\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp\" (UID: \"6d334514-92f2-4d83-8791-4892bf569912\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:55.898174 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:55.898142 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:03:56.176234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:56.176204 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp"] Sep 11 14:03:56.178075 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:03:56.178052 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d334514_92f2_4d83_8791_4892bf569912.slice/crio-0769c01c0d1d824b5beb719b22a03bab9aebc2667fb75ca522b7879b3e91ad89 WatchSource:0}: Error finding container 0769c01c0d1d824b5beb719b22a03bab9aebc2667fb75ca522b7879b3e91ad89: Status 404 returned error can't find the container with id 0769c01c0d1d824b5beb719b22a03bab9aebc2667fb75ca522b7879b3e91ad89 Sep 11 14:03:56.179427 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:56.179406 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:03:57.132917 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:57.132886 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" event={"ID":"6d334514-92f2-4d83-8791-4892bf569912","Type":"ContainerStarted","Data":"2d3ead475cfbb170d7a3a442664cb5f17652864fceb3a92e190241c704440c24"} Sep 11 14:03:57.133344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:57.132936 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" event={"ID":"6d334514-92f2-4d83-8791-4892bf569912","Type":"ContainerStarted","Data":"0769c01c0d1d824b5beb719b22a03bab9aebc2667fb75ca522b7879b3e91ad89"} Sep 11 14:03:57.162561 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:03:57.162506 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" podStartSLOduration=1.955525384 podStartE2EDuration="2.162487712s" totalImagesPullingTime="206.962328ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:03:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:03:57.159496 +0000 UTC m=+654.029227274" watchObservedRunningTime="2026-09-11 14:03:57.162487712 +0000 UTC m=+654.032218974" Sep 11 14:04:04.709317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.709284 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s9996"] Sep 11 14:04:04.711802 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.711780 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.714946 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.714924 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Sep 11 14:04:04.724994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.724967 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s9996"] Sep 11 14:04:04.767062 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.767017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:04.826673 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.826639 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.826833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.826709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.826833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.826743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172bdcd6-0212-4e76-b7ec-846d1fcf0313-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.826914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.826827 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2tqd\" (UniqueName: \"kubernetes.io/projected/172bdcd6-0212-4e76-b7ec-846d1fcf0313-kube-api-access-r2tqd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.826914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.826886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.826914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.826910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.927676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.927640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.927866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.927693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.927866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.927734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172bdcd6-0212-4e76-b7ec-846d1fcf0313-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.927866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.927771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2tqd\" (UniqueName: \"kubernetes.io/projected/172bdcd6-0212-4e76-b7ec-846d1fcf0313-kube-api-access-r2tqd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.927866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.927810 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.927866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.927835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.928177 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.928153 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.928248 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.928184 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.928298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.928248 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.930041 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.930015 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/172bdcd6-0212-4e76-b7ec-846d1fcf0313-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.930352 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.930336 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/172bdcd6-0212-4e76-b7ec-846d1fcf0313-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:04.941865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:04.941845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2tqd\" (UniqueName: \"kubernetes.io/projected/172bdcd6-0212-4e76-b7ec-846d1fcf0313-kube-api-access-r2tqd\") pod \"e2e-trlp-test-simulated-kserve-598466548d-s9996\" (UID: \"172bdcd6-0212-4e76-b7ec-846d1fcf0313\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:05.023131 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:05.023034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:05.297057 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:05.297034 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-s9996"] Sep 11 14:04:05.298829 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:04:05.298798 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172bdcd6_0212_4e76_b7ec_846d1fcf0313.slice/crio-a00d25e394a54dae9934a42c8de93517ad6c344fd95067f3e021b04e109ac8f5 WatchSource:0}: Error finding container a00d25e394a54dae9934a42c8de93517ad6c344fd95067f3e021b04e109ac8f5: Status 404 returned error can't find the container with id a00d25e394a54dae9934a42c8de93517ad6c344fd95067f3e021b04e109ac8f5 Sep 11 14:04:05.898479 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:05.898382 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:04:05.898479 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:05.898456 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:04:05.911954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:05.911926 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:04:06.185320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:06.185232 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" event={"ID":"172bdcd6-0212-4e76-b7ec-846d1fcf0313","Type":"ContainerStarted","Data":"3007610f38c8cff974a2039a084abf0eaa17ceb870a006cce7dbb60d91a87877"} Sep 11 14:04:06.185320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:06.185295 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" event={"ID":"172bdcd6-0212-4e76-b7ec-846d1fcf0313","Type":"ContainerStarted","Data":"a00d25e394a54dae9934a42c8de93517ad6c344fd95067f3e021b04e109ac8f5"} Sep 11 14:04:06.196778 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:06.196753 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp" Sep 11 14:04:06.210159 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:06.210096 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" podStartSLOduration=2.011789068 podStartE2EDuration="2.210079238s" totalImagesPullingTime="198.29017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:04:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:04:06.2080215 +0000 UTC m=+663.077752762" watchObservedRunningTime="2026-09-11 14:04:06.210079238 +0000 UTC m=+663.079810506" Sep 11 14:04:14.107737 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:14.107704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:15.023558 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:15.023491 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:15.023736 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:15.023601 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:15.036620 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:15.036593 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:15.253170 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:15.253144 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-s9996" Sep 11 14:04:27.850946 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:27.850891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:43.710991 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:43.710954 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:43.787286 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:43.787247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:48.344488 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:48.344457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:57.441863 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:57.441830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:04:57.543120 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:04:57.543065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:28.685632 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:05:28.685592 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:28.786208 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:05:28.786172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:05:32.623960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:05:32.623919 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:08.367330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:08.367288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:11.027327 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:11.027288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:11.253608 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:11.253575 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:12.141525 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:12.141493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:12.339406 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:12.339369 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:28.233331 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.233241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:28.453966 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.453934 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:28.764880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.764843 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9"] Sep 11 14:06:28.768326 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.768305 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.768837 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.768801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:28.774022 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.773997 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-abfca2-openshift-default-dockercfg-gc44v\"" Sep 11 14:06:28.802983 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.802951 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9"] Sep 11 14:06:28.906890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.906857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-data\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.906897 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-envoy\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.906923 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-certs\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.906948 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-podinfo\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.907024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwv5q\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-kube-api-access-pwv5q\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.907067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-socket\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.907092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-token\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907408 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.907153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6add6870-c5dd-426f-bf7e-07d762ae299f-istiod-ca-cert\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:28.907408 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:28.907222 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-credential-socket\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008043 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-credential-socket\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008266 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008067 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-data\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008266 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008116 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-envoy\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008266 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-certs\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008266 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-podinfo\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008266 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008224 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwv5q\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-kube-api-access-pwv5q\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008266 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-socket\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008591 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008279 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-token\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008591 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008328 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6add6870-c5dd-426f-bf7e-07d762ae299f-istiod-ca-cert\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008591 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-credential-socket\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008591 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008520 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-data\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008812 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008644 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-certs\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.008812 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.008787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-socket\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.009064 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.009044 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6add6870-c5dd-426f-bf7e-07d762ae299f-istiod-ca-cert\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.010616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.010599 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-envoy\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.010738 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.010717 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-podinfo\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.233835 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.233803 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-token\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.435950 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.435909 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwv5q\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-kube-api-access-pwv5q\") pod \"e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.679614 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.679583 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:29.721513 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:29.721478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:30.071880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:30.071852 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9"] Sep 11 14:06:30.073831 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:06:30.073805 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6add6870_c5dd_426f_bf7e_07d762ae299f.slice/crio-4628420af3ca78bfb0f7b43919b3809e2488ea9a1eecd8a267069b48c3196844 WatchSource:0}: Error finding container 4628420af3ca78bfb0f7b43919b3809e2488ea9a1eecd8a267069b48c3196844: Status 404 returned error can't find the container with id 4628420af3ca78bfb0f7b43919b3809e2488ea9a1eecd8a267069b48c3196844 Sep 11 14:06:30.076158 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:30.076085 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:30.076261 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:30.076199 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:30.076261 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:30.076231 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:30.322086 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:30.321981 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:30.704547 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:30.704457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:31.035220 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:31.035129 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" event={"ID":"6add6870-c5dd-426f-bf7e-07d762ae299f","Type":"ContainerStarted","Data":"06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1"} Sep 11 14:06:31.035392 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:31.035225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" event={"ID":"6add6870-c5dd-426f-bf7e-07d762ae299f","Type":"ContainerStarted","Data":"4628420af3ca78bfb0f7b43919b3809e2488ea9a1eecd8a267069b48c3196844"} Sep 11 14:06:31.104898 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:31.104841 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podStartSLOduration=3.104826094 podStartE2EDuration="3.104826094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:31.103323678 +0000 UTC m=+807.973054964" watchObservedRunningTime="2026-09-11 14:06:31.104826094 +0000 UTC m=+807.974557355" Sep 11 14:06:31.130547 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:31.130521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:31.679969 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:31.679936 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:31.684844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:31.684824 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:32.041570 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:32.041490 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:32.042453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:32.042434 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:33.612366 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:33.612335 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:33.976875 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:33.976782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:35.431459 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:35.431423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:35.597330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:35.597297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:36.936391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:36.936350 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:37.334754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.334714 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:37.582286 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.582254 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm"] Sep 11 14:06:37.585333 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.585269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.589066 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.589041 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-abfca2\"" Sep 11 14:06:37.590526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.590506 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-abfca2-dockercfg-fdlrk\"" Sep 11 14:06:37.620922 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.620900 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm"] Sep 11 14:06:37.695954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.695927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c7e3c-a5dc-4526-9044-f937e546d114-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.696085 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.695974 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpkpm\" (UniqueName: \"kubernetes.io/projected/f12c7e3c-a5dc-4526-9044-f937e546d114-kube-api-access-bpkpm\") pod \"payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.797672 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.797633 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpkpm\" (UniqueName: \"kubernetes.io/projected/f12c7e3c-a5dc-4526-9044-f937e546d114-kube-api-access-bpkpm\") pod \"payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.797844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.797827 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c7e3c-a5dc-4526-9044-f937e546d114-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.798528 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.798511 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c7e3c-a5dc-4526-9044-f937e546d114-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.807784 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.807762 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpkpm\" (UniqueName: \"kubernetes.io/projected/f12c7e3c-a5dc-4526-9044-f937e546d114-kube-api-access-bpkpm\") pod \"payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:37.896549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:37.896480 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:38.215984 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:38.215956 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm"] Sep 11 14:06:38.217524 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:06:38.217493 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12c7e3c_a5dc_4526_9044_f937e546d114.slice/crio-fd0f3d92d34def5d64db4923c3f551e4ac268b8a805596f32cb1262328d4708d WatchSource:0}: Error finding container fd0f3d92d34def5d64db4923c3f551e4ac268b8a805596f32cb1262328d4708d: Status 404 returned error can't find the container with id fd0f3d92d34def5d64db4923c3f551e4ac268b8a805596f32cb1262328d4708d Sep 11 14:06:38.914697 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:38.914657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:39.084779 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.084731 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" event={"ID":"f12c7e3c-a5dc-4526-9044-f937e546d114","Type":"ContainerStarted","Data":"f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7"} Sep 11 14:06:39.084929 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.084797 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:39.084929 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.084819 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" event={"ID":"f12c7e3c-a5dc-4526-9044-f937e546d114","Type":"ContainerStarted","Data":"fd0f3d92d34def5d64db4923c3f551e4ac268b8a805596f32cb1262328d4708d"} Sep 11 14:06:39.146032 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.145970 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" podStartSLOduration=2.145946916 podStartE2EDuration="2.145946916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:39.144415559 +0000 UTC m=+816.014146822" watchObservedRunningTime="2026-09-11 14:06:39.145946916 +0000 UTC m=+816.015678180" Sep 11 14:06:39.318815 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.318784 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f"] Sep 11 14:06:39.321646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.321622 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.418495 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:06:39.418459 2579 status_manager.go:1018] "Failed to get status for pod" err="pods \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" is forbidden: User \"system:node:ip-10-0-136-162.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-136-162.ec2.internal' and this object" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.420830 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.420803 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-0bd77b-openshift-default-dockercfg-pgbxk\"" Sep 11 14:06:39.495375 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.495337 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f"] Sep 11 14:06:39.515852 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.515822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hbdp\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-kube-api-access-4hbdp\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.515990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.515859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-credential-socket\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.515990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.515883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-data\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.515990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.515947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-podinfo\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.515990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.515963 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-certs\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.516167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.516017 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-envoy\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.516167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.516038 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ca79ba-0156-4783-a157-5bf4c53ee5db-istiod-ca-cert\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.516167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.516095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-socket\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.516267 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.516174 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-token\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.616978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.616847 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-token\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.616978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.616903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hbdp\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-kube-api-access-4hbdp\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.616978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.616939 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-credential-socket\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.616978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.616972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-data\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-podinfo\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617047 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-certs\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-envoy\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617148 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ca79ba-0156-4783-a157-5bf4c53ee5db-istiod-ca-cert\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-socket\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617780 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-credential-socket\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617796 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-data\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.617868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.617821 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-certs\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.618198 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.618005 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-socket\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.618320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.618302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ca79ba-0156-4783-a157-5bf4c53ee5db-istiod-ca-cert\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.619711 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.619689 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-envoy\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.619841 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.619825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-podinfo\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.639747 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.639718 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-token\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.648459 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.648429 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hbdp\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-kube-api-access-4hbdp\") pod \"e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:39.706043 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.705999 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:39.932210 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:39.932135 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:40.123418 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:40.123387 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:40.313427 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:06:40.313404 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ca79ba_0156_4783_a157_5bf4c53ee5db.slice/crio-a55fbc0ed03a25d4b4e9eb084c35c79e633cf6c1ed08a5a2367fef458599abfc WatchSource:0}: Error finding container a55fbc0ed03a25d4b4e9eb084c35c79e633cf6c1ed08a5a2367fef458599abfc: Status 404 returned error can't find the container with id a55fbc0ed03a25d4b4e9eb084c35c79e633cf6c1ed08a5a2367fef458599abfc Sep 11 14:06:40.315087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:40.315060 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f"] Sep 11 14:06:40.315826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:40.315796 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:40.315908 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:40.315855 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:40.315908 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:40.315882 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:41.097718 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:41.097678 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" event={"ID":"58ca79ba-0156-4783-a157-5bf4c53ee5db","Type":"ContainerStarted","Data":"8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e"} Sep 11 14:06:41.098149 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:41.097738 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" event={"ID":"58ca79ba-0156-4783-a157-5bf4c53ee5db","Type":"ContainerStarted","Data":"a55fbc0ed03a25d4b4e9eb084c35c79e633cf6c1ed08a5a2367fef458599abfc"} Sep 11 14:06:41.225910 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:41.225832 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podStartSLOduration=2.225806215 podStartE2EDuration="2.225806215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:41.224902225 +0000 UTC m=+818.094633506" watchObservedRunningTime="2026-09-11 14:06:41.225806215 +0000 UTC m=+818.095537478" Sep 11 14:06:41.932348 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:41.932315 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:41.938342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:41.938317 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:42.103790 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:42.103757 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:42.104910 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:42.104881 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:06:42.877831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:42.877743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:43.574830 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:43.574793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:43.717561 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:43.717530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:44.161905 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:44.161871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:44.573928 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:44.573888 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:44.964895 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:44.964814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:45.069729 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:45.069697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:45.184643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:45.184611 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:46.044908 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:46.044878 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:46.198474 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:46.198439 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:46.303746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:46.303656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:46.473920 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:46.473872 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:46.923399 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:46.923359 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:47.086948 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:47.086891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:47.333800 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:47.333767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:47.736558 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:47.736457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:47.862835 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:47.862781 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:48.119363 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.119329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:48.433784 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.433711 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n"] Sep 11 14:06:48.436391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.436373 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.439230 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.439211 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-80fb45-openshift-default-dockercfg-rqd8f\"" Sep 11 14:06:48.459502 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.459471 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n"] Sep 11 14:06:48.519453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519416 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istiod-ca-cert\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519511 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-certs\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-podinfo\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519756 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519592 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-token\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519756 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519682 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-credential-socket\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519756 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519733 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-envoy\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519909 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpbg8\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-kube-api-access-qpbg8\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519909 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519783 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-data\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.519909 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.519822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-socket\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.550083 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.550046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:48.620497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-certs\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-podinfo\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620531 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-token\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-credential-socket\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620616 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-envoy\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620642 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpbg8\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-kube-api-access-qpbg8\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-data\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.620725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.620719 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-socket\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.621055 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.621006 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-credential-socket\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.621145 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.621063 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istiod-ca-cert\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.621145 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.621089 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-data\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.621275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.621175 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-socket\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.621336 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.621320 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-certs\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.621781 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.621752 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istiod-ca-cert\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.623090 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.623062 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-envoy\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.623226 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.623210 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-podinfo\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.638043 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.638013 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpbg8\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-kube-api-access-qpbg8\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.639152 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.639090 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-token\") pod \"e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.743798 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.743707 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:48.748742 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.748710 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:48.975567 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:48.975531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:49.039790 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.039709 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n"] Sep 11 14:06:49.047366 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:06:49.041768 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d15ad8_53e7_463b_a8d4_a96896c1e6c3.slice/crio-375ee73a11afe639c681c89552cc4ddc171673144180c1e0c47c2bb4ac358d61 WatchSource:0}: Error finding container 375ee73a11afe639c681c89552cc4ddc171673144180c1e0c47c2bb4ac358d61: Status 404 returned error can't find the container with id 375ee73a11afe639c681c89552cc4ddc171673144180c1e0c47c2bb4ac358d61 Sep 11 14:06:49.047366 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.043839 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:49.047366 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.043924 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:49.047366 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.043966 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:49.149146 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.149072 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" event={"ID":"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3","Type":"ContainerStarted","Data":"a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8"} Sep 11 14:06:49.149486 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.149167 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" event={"ID":"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3","Type":"ContainerStarted","Data":"375ee73a11afe639c681c89552cc4ddc171673144180c1e0c47c2bb4ac358d61"} Sep 11 14:06:49.178434 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.178369 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podStartSLOduration=1.178345017 podStartE2EDuration="1.178345017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:49.173181314 +0000 UTC m=+826.042912580" watchObservedRunningTime="2026-09-11 14:06:49.178345017 +0000 UTC m=+826.048076280" Sep 11 14:06:49.694220 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.694182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:49.749833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.749806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:49.855858 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.855808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:49.974079 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:49.973994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:50.084010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:50.083969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:50.093456 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:50.093361 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:06:50.357616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:50.357553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:50.745406 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:50.745321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:50.756472 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:50.756447 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:50.838954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:50.838923 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:51.162508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:51.162477 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:51.163509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:51.163485 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:06:51.359231 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:51.359194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:51.456706 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:51.456676 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:52.169696 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.169656 2579 generic.go:356] "Generic (PLEG): container finished" podID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerID="06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1" exitCode=0 Sep 11 14:06:52.169896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.169729 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" event={"ID":"6add6870-c5dd-426f-bf7e-07d762ae299f","Type":"ContainerDied","Data":"06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1"} Sep 11 14:06:52.170347 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.170319 2579 scope.go:117] "RemoveContainer" containerID="06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1" Sep 11 14:06:52.171364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.171337 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:52.171473 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.171402 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:52.171473 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.171452 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:06:52.528432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.528326 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:52.705675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:52.705643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:53.183696 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.183655 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" event={"ID":"6add6870-c5dd-426f-bf7e-07d762ae299f","Type":"ContainerStarted","Data":"33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3"} Sep 11 14:06:53.449999 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.449910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:53.550996 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.550950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:53.679912 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.679876 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:53.680383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.680352 2579 patch_prober.go:28] interesting pod/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 11 14:06:53.680490 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.680419 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 11 14:06:53.714957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.714855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:53.828804 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.828764 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:53.945037 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:53.945003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:54.071773 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.071741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:54.168298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.168266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:54.243086 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.243058 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl"] Sep 11 14:06:54.245602 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.245585 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.249437 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.249417 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-80fb45-dockercfg-rrgtr\"" Sep 11 14:06:54.249663 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.249644 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-80fb45\"" Sep 11 14:06:54.257817 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.257794 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl"] Sep 11 14:06:54.396967 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.396871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf9nq\" (UniqueName: \"kubernetes.io/projected/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-kube-api-access-cf9nq\") pod \"payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.397132 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.397044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.497784 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.497740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.497971 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.497809 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf9nq\" (UniqueName: \"kubernetes.io/projected/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-kube-api-access-cf9nq\") pod \"payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.498555 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.498536 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.507337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.507315 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf9nq\" (UniqueName: \"kubernetes.io/projected/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-kube-api-access-cf9nq\") pod \"payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.526596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.526566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:54.557017 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.556992 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:54.681893 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.681406 2579 patch_prober.go:28] interesting pod/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 11 14:06:54.681893 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.681465 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 11 14:06:54.702454 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.702406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:54.855564 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:54.855517 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl"] Sep 11 14:06:54.856745 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:06:54.856719 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb2b5b62_cfe5_4e6d_b0bf_84d2033988cd.slice/crio-6bc829550f20a11383da6873e05bd546703c0f0f47f74327d556cb3baf7a5371 WatchSource:0}: Error finding container 6bc829550f20a11383da6873e05bd546703c0f0f47f74327d556cb3baf7a5371: Status 404 returned error can't find the container with id 6bc829550f20a11383da6873e05bd546703c0f0f47f74327d556cb3baf7a5371 Sep 11 14:06:55.203398 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.203300 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" event={"ID":"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd","Type":"ContainerStarted","Data":"60a6ad5a23c97618344888fd5c8508ebfe472a73b4df8333045710808c3d4816"} Sep 11 14:06:55.203398 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.203365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" event={"ID":"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd","Type":"ContainerStarted","Data":"6bc829550f20a11383da6873e05bd546703c0f0f47f74327d556cb3baf7a5371"} Sep 11 14:06:55.203642 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.203555 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:06:55.224509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.224461 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" podStartSLOduration=1.224446986 podStartE2EDuration="1.224446986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:55.222066274 +0000 UTC m=+832.091797537" watchObservedRunningTime="2026-09-11 14:06:55.224446986 +0000 UTC m=+832.094178248" Sep 11 14:06:55.685167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.683504 2579 patch_prober.go:28] interesting pod/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 11 14:06:55.685167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.683585 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 11 14:06:55.729044 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.728992 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:55.849616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:55.849582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.020780 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.020695 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.389136 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.388115 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.490993 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.490962 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.684198 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.681291 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r"] Sep 11 14:06:56.690139 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.689258 2579 patch_prober.go:28] interesting pod/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 11 14:06:56.690139 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.689329 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 11 14:06:56.708227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.706343 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r"] Sep 11 14:06:56.708227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.706464 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:56.709263 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.709243 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-4e6230ad-dockercfg-2vgsw\"" Sep 11 14:06:56.709555 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.709289 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-4e6230ad\"" Sep 11 14:06:56.823169 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.823139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9n96\" (UniqueName: \"kubernetes.io/projected/19585af8-1f2f-4254-bd1a-7b1143f9c731-kube-api-access-k9n96\") pod \"payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:56.823304 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.823213 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19585af8-1f2f-4254-bd1a-7b1143f9c731-plugins-config-volume\") pod \"payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:56.914789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.914758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:56.923954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.923932 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9n96\" (UniqueName: \"kubernetes.io/projected/19585af8-1f2f-4254-bd1a-7b1143f9c731-kube-api-access-k9n96\") pod \"payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:56.924155 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.924142 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19585af8-1f2f-4254-bd1a-7b1143f9c731-plugins-config-volume\") pod \"payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:56.924830 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.924811 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19585af8-1f2f-4254-bd1a-7b1143f9c731-plugins-config-volume\") pod \"payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:56.934876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:56.934826 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9n96\" (UniqueName: \"kubernetes.io/projected/19585af8-1f2f-4254-bd1a-7b1143f9c731-kube-api-access-k9n96\") pod \"payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:57.025311 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:57.025280 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:57.352168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:57.352142 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r"] Sep 11 14:06:57.353769 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:06:57.353748 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19585af8_1f2f_4254_bd1a_7b1143f9c731.slice/crio-ba817eb28727f0262eca9a0b36a19bc22d59a10ad319ae336c125e35d537479c WatchSource:0}: Error finding container ba817eb28727f0262eca9a0b36a19bc22d59a10ad319ae336c125e35d537479c: Status 404 returned error can't find the container with id ba817eb28727f0262eca9a0b36a19bc22d59a10ad319ae336c125e35d537479c Sep 11 14:06:57.680343 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:57.680257 2579 patch_prober.go:28] interesting pod/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Sep 11 14:06:57.680613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:57.680579 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Sep 11 14:06:57.802954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:57.802868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:58.074247 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.074216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:58.228643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.228594 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" event={"ID":"19585af8-1f2f-4254-bd1a-7b1143f9c731","Type":"ContainerStarted","Data":"b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb"} Sep 11 14:06:58.228821 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.228671 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" event={"ID":"19585af8-1f2f-4254-bd1a-7b1143f9c731","Type":"ContainerStarted","Data":"ba817eb28727f0262eca9a0b36a19bc22d59a10ad319ae336c125e35d537479c"} Sep 11 14:06:58.229566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.229535 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:06:58.265819 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.265756 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" podStartSLOduration=2.265731491 podStartE2EDuration="2.265731491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:06:58.26527457 +0000 UTC m=+835.135005832" watchObservedRunningTime="2026-09-11 14:06:58.265731491 +0000 UTC m=+835.135462753" Sep 11 14:06:58.755789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.755755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:06:58.863676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.863632 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:58.864989 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.864923 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:58.869549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:58.869522 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:06:59.600131 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:06:59.600064 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:00.424876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:00.424841 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.483721 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:01.483687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.608953 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:01.608916 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.690749 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:01.690706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:01.818080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:01.818046 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:06.212387 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:06.212344 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:07:06.982881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:06.981864 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:57644->10.134.0.55:15021: read: connection reset by peer" start-of-body= Sep 11 14:07:06.982881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:06.981943 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:57644->10.134.0.55:15021: read: connection reset by peer" Sep 11 14:07:07.291289 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.291208 2579 generic.go:356] "Generic (PLEG): container finished" podID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerID="8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e" exitCode=0 Sep 11 14:07:07.291694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.291288 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" event={"ID":"58ca79ba-0156-4783-a157-5bf4c53ee5db","Type":"ContainerDied","Data":"8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e"} Sep 11 14:07:07.291810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.291795 2579 scope.go:117] "RemoveContainer" containerID="8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e" Sep 11 14:07:07.292919 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.292893 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:07.292999 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.292969 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:07.293047 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.293000 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:07.293114 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.293082 2579 generic.go:356] "Generic (PLEG): container finished" podID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerID="a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8" exitCode=0 Sep 11 14:07:07.293175 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.293137 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" event={"ID":"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3","Type":"ContainerDied","Data":"a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8"} Sep 11 14:07:07.293475 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.293462 2579 scope.go:117] "RemoveContainer" containerID="a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8" Sep 11 14:07:07.294538 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.294510 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:07.294630 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.294570 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:07.294630 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.294601 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:07.685067 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:07.685032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:08.307072 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.307027 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" event={"ID":"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3","Type":"ContainerStarted","Data":"698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24"} Sep 11 14:07:08.310960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.310933 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" event={"ID":"58ca79ba-0156-4783-a157-5bf4c53ee5db","Type":"ContainerStarted","Data":"4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e"} Sep 11 14:07:08.749954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.749873 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:07:08.750236 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.749986 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:07:08.750345 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.750290 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 14:07:08.932879 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.932850 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:07:08.933873 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.933843 2579 patch_prober.go:28] interesting pod/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:07:08.934071 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.934047 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 11 14:07:08.954314 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:08.954280 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:09.020978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:09.020823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:09.749226 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:09.749196 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:07:09.749671 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:09.749645 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 14:07:09.932755 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:09.932725 2579 patch_prober.go:28] interesting pod/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:07:09.932990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:09.932969 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 11 14:07:10.247363 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:10.247335 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:07:10.750145 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:10.750111 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:07:10.750527 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:10.750166 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 14:07:10.933322 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:10.933284 2579 patch_prober.go:28] interesting pod/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:07:10.933505 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:10.933343 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 11 14:07:11.749433 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:11.749399 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:07:11.749612 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:11.749460 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 14:07:11.932401 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:11.932366 2579 patch_prober.go:28] interesting pod/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:07:11.932769 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:11.932415 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 11 14:07:12.105250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:12.104271 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:07:12.749765 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:12.749726 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:07:12.749945 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:12.749791 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 14:07:12.933093 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:12.933059 2579 patch_prober.go:28] interesting pod/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Sep 11 14:07:12.933499 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:12.933144 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Sep 11 14:07:13.749400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:13.749369 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Sep 11 14:07:13.749567 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:13.749425 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Sep 11 14:07:13.937851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:13.937806 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:07:13.938789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:13.938769 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:07:14.609333 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:14.609298 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:14.752890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:14.752857 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:07:14.753144 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:14.753127 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:07:14.753908 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:14.753888 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:07:15.812933 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:15.812905 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:16.295834 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:16.295802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:16.856488 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:16.856457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:17.971301 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:17.971271 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:18.826734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:18.826706 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:19.564468 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:19.564430 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:20.440255 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:20.440213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:21.537292 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:21.537249 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:22.339618 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:22.339583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:23.074483 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:23.074448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:23.932332 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:23.932296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:24.648395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:24.648357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:25.444853 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:25.444819 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:29.827064 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:29.827031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:32.348146 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:32.348090 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.140484 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.140446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.187030 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.187000 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r"] Sep 11 14:07:33.187250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.187225 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" podUID="19585af8-1f2f-4254-bd1a-7b1143f9c731" containerName="payload-processing" containerID="cri-o://b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb" gracePeriod=30 Sep 11 14:07:33.229638 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.229606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.336767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.336732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.465667 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.465639 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.468734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.468715 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:07:33.474602 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.474576 2579 generic.go:356] "Generic (PLEG): container finished" podID="19585af8-1f2f-4254-bd1a-7b1143f9c731" containerID="b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb" exitCode=0 Sep 11 14:07:33.474704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.474613 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" Sep 11 14:07:33.474704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.474654 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" event={"ID":"19585af8-1f2f-4254-bd1a-7b1143f9c731","Type":"ContainerDied","Data":"b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb"} Sep 11 14:07:33.474775 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.474710 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r" event={"ID":"19585af8-1f2f-4254-bd1a-7b1143f9c731","Type":"ContainerDied","Data":"ba817eb28727f0262eca9a0b36a19bc22d59a10ad319ae336c125e35d537479c"} Sep 11 14:07:33.474775 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.474741 2579 scope.go:117] "RemoveContainer" containerID="b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb" Sep 11 14:07:33.484607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.484586 2579 scope.go:117] "RemoveContainer" containerID="b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb" Sep 11 14:07:33.484864 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:07:33.484844 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb\": container with ID starting with b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb not found: ID does not exist" containerID="b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb" Sep 11 14:07:33.484926 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.484875 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb"} err="failed to get container status \"b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb\": rpc error: code = NotFound desc = could not find container \"b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb\": container with ID starting with b4254ed6379bf41eb0acba3033986b998aadee138287a67a12dc35aedda4ffdb not found: ID does not exist" Sep 11 14:07:33.605027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.604990 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:33.624048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.624031 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9n96\" (UniqueName: \"kubernetes.io/projected/19585af8-1f2f-4254-bd1a-7b1143f9c731-kube-api-access-k9n96\") pod \"19585af8-1f2f-4254-bd1a-7b1143f9c731\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " Sep 11 14:07:33.624177 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.624070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19585af8-1f2f-4254-bd1a-7b1143f9c731-plugins-config-volume\") pod \"19585af8-1f2f-4254-bd1a-7b1143f9c731\" (UID: \"19585af8-1f2f-4254-bd1a-7b1143f9c731\") " Sep 11 14:07:33.624474 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.624448 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19585af8-1f2f-4254-bd1a-7b1143f9c731-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "19585af8-1f2f-4254-bd1a-7b1143f9c731" (UID: "19585af8-1f2f-4254-bd1a-7b1143f9c731"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:07:33.626447 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.626425 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19585af8-1f2f-4254-bd1a-7b1143f9c731-kube-api-access-k9n96" (OuterVolumeSpecName: "kube-api-access-k9n96") pod "19585af8-1f2f-4254-bd1a-7b1143f9c731" (UID: "19585af8-1f2f-4254-bd1a-7b1143f9c731"). InnerVolumeSpecName "kube-api-access-k9n96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:07:33.725704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.725599 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9n96\" (UniqueName: \"kubernetes.io/projected/19585af8-1f2f-4254-bd1a-7b1143f9c731-kube-api-access-k9n96\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:07:33.725704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.725636 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19585af8-1f2f-4254-bd1a-7b1143f9c731-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:07:33.800962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.800932 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r"] Sep 11 14:07:33.810275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.810240 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-4e6230ad-596547944f-qpg6r"] Sep 11 14:07:33.994967 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:33.994886 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:34.217037 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.217000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:34.343562 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.343532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:34.590411 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.590380 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h"] Sep 11 14:07:34.591214 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.591179 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19585af8-1f2f-4254-bd1a-7b1143f9c731" containerName="payload-processing" Sep 11 14:07:34.591214 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.591205 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="19585af8-1f2f-4254-bd1a-7b1143f9c731" containerName="payload-processing" Sep 11 14:07:34.591399 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.591308 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="19585af8-1f2f-4254-bd1a-7b1143f9c731" containerName="payload-processing" Sep 11 14:07:34.594477 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.594419 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.597069 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.597047 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-42670434-gw-openshift-default-dockercfg-h6v58\"" Sep 11 14:07:34.608991 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.608963 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h"] Sep 11 14:07:34.623789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.623763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:34.736603 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.736566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-podinfo\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.736603 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.736605 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-certs\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.736806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.736625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-kube-api-access-gx6ct\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.736806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.736747 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-socket\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.736806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.736780 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istiod-ca-cert\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.736965 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.736947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-token\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.737195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.737165 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-envoy\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.737330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.737216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-credential-socket\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.737330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.737245 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-data\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.760495 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.760471 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:34.838685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-socket\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istiod-ca-cert\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838730 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-token\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838826 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-envoy\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838861 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-credential-socket\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-data\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-podinfo\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.838992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.838958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-certs\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.839344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.839266 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-credential-socket\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.839344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.839320 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-kube-api-access-gx6ct\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.839461 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.839340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-data\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.839461 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.839394 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-certs\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.839563 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.839500 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-socket\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.839602 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.839572 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istiod-ca-cert\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.841240 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.841220 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-envoy\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.841566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.841549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-podinfo\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.849035 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.848982 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-token\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.851173 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.851155 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-kube-api-access-gx6ct\") pod \"e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:34.905959 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:34.905932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:35.190681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.189008 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h"] Sep 11 14:07:35.196530 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.196499 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:35.196642 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.196574 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:35.196642 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.196604 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:35.230346 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.230312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:35.490163 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.490056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" event={"ID":"4b5cb804-7c0a-4896-b72a-1e0e2556b970","Type":"ContainerStarted","Data":"c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f"} Sep 11 14:07:35.490163 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.490136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" event={"ID":"4b5cb804-7c0a-4896-b72a-1e0e2556b970","Type":"ContainerStarted","Data":"0838dc4e7670be6a2f28baadf29d73f1d2fe1984106b74b5e3cce08f58957a01"} Sep 11 14:07:35.522202 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.522147 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podStartSLOduration=1.5221276750000001 podStartE2EDuration="1.522127675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:07:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:07:35.520601456 +0000 UTC m=+872.390332716" watchObservedRunningTime="2026-09-11 14:07:35.522127675 +0000 UTC m=+872.391858951" Sep 11 14:07:35.649509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.649471 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19585af8-1f2f-4254-bd1a-7b1143f9c731" path="/var/lib/kubelet/pods/19585af8-1f2f-4254-bd1a-7b1143f9c731/volumes" Sep 11 14:07:35.906576 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:35.906541 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:36.244557 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:36.244461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:36.912200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:36.912166 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:37.089530 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:37.089493 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:37.192982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:37.192882 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:37.504871 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:37.504772 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:37.505876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:37.505851 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:37.637517 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:37.637476 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:38.640608 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:38.640562 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:38.747565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:38.747529 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:38.877276 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:38.877241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:39.423087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:39.423039 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:39.532497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:39.532464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:40.154249 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:40.154213 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:41.145015 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:41.144986 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:42.366379 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:42.366342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:43.559440 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:43.559408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:57.240388 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.240294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:07:57.646638 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.646607 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerID="c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f" exitCode=0 Sep 11 14:07:57.646810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.646687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" event={"ID":"4b5cb804-7c0a-4896-b72a-1e0e2556b970","Type":"ContainerDied","Data":"c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f"} Sep 11 14:07:57.647114 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.647085 2579 scope.go:117] "RemoveContainer" containerID="c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f" Sep 11 14:07:57.648250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.648225 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:57.648324 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.648290 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:57.648364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:57.648323 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:07:58.655637 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:58.655608 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" event={"ID":"4b5cb804-7c0a-4896-b72a-1e0e2556b970","Type":"ContainerStarted","Data":"10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842"} Sep 11 14:07:58.907209 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:58.907088 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:07:58.907362 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:58.907276 2579 patch_prober.go:28] interesting pod/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:07:58.907362 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:58.907324 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 14:07:59.906446 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:59.906415 2579 patch_prober.go:28] interesting pod/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:07:59.906839 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:07:59.906473 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 14:08:00.164756 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:00.164668 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:00.906621 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:00.906585 2579 patch_prober.go:28] interesting pod/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:08:00.907207 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:00.906648 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 14:08:01.906815 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:01.906774 2579 patch_prober.go:28] interesting pod/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:08:01.907240 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:01.906853 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 14:08:02.001920 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.001887 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp"] Sep 11 14:08:02.004637 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.004621 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:02.007352 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.007327 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:08:02.007484 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.007366 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:08:02.016072 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.016049 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp"] Sep 11 14:08:02.129835 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.129800 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47809b02-593d-4bf2-90db-879e3c1d2848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-42670434-7z9xp\" (UID: \"47809b02-593d-4bf2-90db-879e3c1d2848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:02.153932 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.153893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:02.230823 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.230734 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47809b02-593d-4bf2-90db-879e3c1d2848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-42670434-7z9xp\" (UID: \"47809b02-593d-4bf2-90db-879e3c1d2848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:02.231446 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.231427 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47809b02-593d-4bf2-90db-879e3c1d2848-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-42670434-7z9xp\" (UID: \"47809b02-593d-4bf2-90db-879e3c1d2848\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:02.316759 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.316714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:02.614648 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.614622 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp"] Sep 11 14:08:02.615730 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:08:02.615699 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47809b02_593d_4bf2_90db_879e3c1d2848.slice/crio-50620dd653d7a45867399c647e916b37b4f294e8a8117508034373fbac570d43 WatchSource:0}: Error finding container 50620dd653d7a45867399c647e916b37b4f294e8a8117508034373fbac570d43: Status 404 returned error can't find the container with id 50620dd653d7a45867399c647e916b37b4f294e8a8117508034373fbac570d43 Sep 11 14:08:02.690453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.690403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" event={"ID":"47809b02-593d-4bf2-90db-879e3c1d2848","Type":"ContainerStarted","Data":"50620dd653d7a45867399c647e916b37b4f294e8a8117508034373fbac570d43"} Sep 11 14:08:02.906896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.906810 2579 patch_prober.go:28] interesting pod/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:08:02.906896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:02.906867 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 14:08:03.454303 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:03.454266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:03.906810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:03.906773 2579 patch_prober.go:28] interesting pod/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Sep 11 14:08:03.907008 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:03.906855 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Sep 11 14:08:04.340988 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:04.340950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:04.911116 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:04.911077 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:08:04.911534 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:04.911507 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:08:04.912202 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:04.912180 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:08:05.451584 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:05.451502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:05.719136 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:05.719008 2579 generic.go:356] "Generic (PLEG): container finished" podID="47809b02-593d-4bf2-90db-879e3c1d2848" containerID="0e117d599925fc2f6480c06ffdbda3bcde202d6971040b78b4e734685ce5059c" exitCode=0 Sep 11 14:08:05.719136 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:05.719091 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" event={"ID":"47809b02-593d-4bf2-90db-879e3c1d2848","Type":"ContainerDied","Data":"0e117d599925fc2f6480c06ffdbda3bcde202d6971040b78b4e734685ce5059c"} Sep 11 14:08:06.629561 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:06.629526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:06.864119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:06.864081 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:06.984057 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:06.983981 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47809b02-593d-4bf2-90db-879e3c1d2848-maas-api-service-ca\") pod \"47809b02-593d-4bf2-90db-879e3c1d2848\" (UID: \"47809b02-593d-4bf2-90db-879e3c1d2848\") " Sep 11 14:08:06.984411 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:06.984391 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47809b02-593d-4bf2-90db-879e3c1d2848-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47809b02-593d-4bf2-90db-879e3c1d2848" (UID: "47809b02-593d-4bf2-90db-879e3c1d2848"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:08:07.085353 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:07.085324 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47809b02-593d-4bf2-90db-879e3c1d2848-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:07.343173 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:07.343130 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:07.555396 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:07.555356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:07.736442 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:07.736403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" event={"ID":"47809b02-593d-4bf2-90db-879e3c1d2848","Type":"ContainerDied","Data":"50620dd653d7a45867399c647e916b37b4f294e8a8117508034373fbac570d43"} Sep 11 14:08:07.736442 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:07.736438 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp" Sep 11 14:08:07.736956 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:07.736466 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50620dd653d7a45867399c647e916b37b4f294e8a8117508034373fbac570d43" Sep 11 14:08:08.311986 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.311930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.423696 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.423661 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.526360 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.526321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.659987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.659892 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.736281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.736249 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp"] Sep 11 14:08:08.749370 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.749339 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-42670434-7z9xp"] Sep 11 14:08:08.788034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.787994 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:08.906283 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.906252 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h"] Sep 11 14:08:08.906480 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.906458 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" containerID="cri-o://10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842" gracePeriod=30 Sep 11 14:08:08.971948 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:08.971855 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.204798 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:09.204768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.434579 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:09.434532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:09.643219 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:09.643187 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47809b02-593d-4bf2-90db-879e3c1d2848" path="/var/lib/kubelet/pods/47809b02-593d-4bf2-90db-879e3c1d2848/volumes" Sep 11 14:08:09.858671 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:09.858640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:10.133417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:10.133320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:10.739221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:10.739172 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:11.985511 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:11.985466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:12.129546 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:12.129502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:12.259463 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:12.259378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:13.388233 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:13.388191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:14.188056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.188030 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:08:14.262026 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.261995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istiod-ca-cert\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262059 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-socket\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262096 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-token\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262158 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-podinfo\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262217 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-certs\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262241 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-data\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262304 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-kube-api-access-gx6ct\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262344 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-envoy\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262376 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:08:14.262451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262409 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-credential-socket\") pod \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\" (UID: \"4b5cb804-7c0a-4896-b72a-1e0e2556b970\") " Sep 11 14:08:14.262724 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262457 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:08:14.262724 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262511 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:08:14.262724 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262538 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-data" (OuterVolumeSpecName: "istio-data") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:08:14.262883 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:08:14.262883 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262853 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.262883 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262874 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.263007 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262890 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.263007 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.262907 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.263130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.263017 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.264560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.264517 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:08:14.264996 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.264972 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-token" (OuterVolumeSpecName: "istio-token") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:14.265128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.265088 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-kube-api-access-gx6ct" (OuterVolumeSpecName: "kube-api-access-gx6ct") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "kube-api-access-gx6ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:08:14.265128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.265094 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4b5cb804-7c0a-4896-b72a-1e0e2556b970" (UID: "4b5cb804-7c0a-4896-b72a-1e0e2556b970"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:08:14.335129 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.335061 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:14.364439 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.364409 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.364439 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.364435 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.364646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.364447 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx6ct\" (UniqueName: \"kubernetes.io/projected/4b5cb804-7c0a-4896-b72a-1e0e2556b970-kube-api-access-gx6ct\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.364646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.364465 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b5cb804-7c0a-4896-b72a-1e0e2556b970-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:14.440140 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.440015 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:14.583540 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.583468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:14.787978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.787900 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerID="10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842" exitCode=0 Sep 11 14:08:14.787978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.787946 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" Sep 11 14:08:14.788193 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.787994 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" event={"ID":"4b5cb804-7c0a-4896-b72a-1e0e2556b970","Type":"ContainerDied","Data":"10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842"} Sep 11 14:08:14.788193 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.788041 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h" event={"ID":"4b5cb804-7c0a-4896-b72a-1e0e2556b970","Type":"ContainerDied","Data":"0838dc4e7670be6a2f28baadf29d73f1d2fe1984106b74b5e3cce08f58957a01"} Sep 11 14:08:14.788193 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.788065 2579 scope.go:117] "RemoveContainer" containerID="10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842" Sep 11 14:08:14.798508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.798490 2579 scope.go:117] "RemoveContainer" containerID="c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f" Sep 11 14:08:14.807186 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.807166 2579 scope.go:117] "RemoveContainer" containerID="10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842" Sep 11 14:08:14.807430 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:08:14.807413 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842\": container with ID starting with 10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842 not found: ID does not exist" containerID="10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842" Sep 11 14:08:14.807494 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.807437 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842"} err="failed to get container status \"10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842\": rpc error: code = NotFound desc = could not find container \"10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842\": container with ID starting with 10da7d13c5ccf385bdbda1c6c7a5a12f21795b4053fe6dec90d9e11e1f095842 not found: ID does not exist" Sep 11 14:08:14.807494 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.807452 2579 scope.go:117] "RemoveContainer" containerID="c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f" Sep 11 14:08:14.807730 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:08:14.807706 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f\": container with ID starting with c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f not found: ID does not exist" containerID="c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f" Sep 11 14:08:14.807817 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.807735 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f"} err="failed to get container status \"c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f\": rpc error: code = NotFound desc = could not find container \"c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f\": container with ID starting with c0d5a24bd1cf78be98f559aa1393261e4ede1acdb1811bec9f386e1d82235c9f not found: ID does not exist" Sep 11 14:08:14.821337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.821313 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h"] Sep 11 14:08:14.826111 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.826079 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-42670434-gw-openshift-default-647fcdc94d-rrd7h"] Sep 11 14:08:14.968845 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:14.968810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:15.127554 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:15.127500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:15.645075 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:15.645047 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" path="/var/lib/kubelet/pods/4b5cb804-7c0a-4896-b72a-1e0e2556b970/volumes" Sep 11 14:08:15.649394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:15.649367 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:16.667151 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:16.667095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:17.548843 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:17.548813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:26.872680 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.872645 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4"] Sep 11 14:08:26.873132 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873116 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47809b02-593d-4bf2-90db-879e3c1d2848" containerName="revoke-keys" Sep 11 14:08:26.873132 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873133 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="47809b02-593d-4bf2-90db-879e3c1d2848" containerName="revoke-keys" Sep 11 14:08:26.873221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873143 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" Sep 11 14:08:26.873221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873148 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" Sep 11 14:08:26.873221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873158 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" Sep 11 14:08:26.873221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873164 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" Sep 11 14:08:26.873349 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873235 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" Sep 11 14:08:26.873349 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.873244 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="47809b02-593d-4bf2-90db-879e3c1d2848" containerName="revoke-keys" Sep 11 14:08:26.876204 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.876187 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:26.879374 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.879353 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0bd77b\"/\"openshift-service-ca.crt\"" Sep 11 14:08:26.879632 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.879612 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0bd77b\"/\"default-dockercfg-cwgsx\"" Sep 11 14:08:26.879764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.879675 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-0bd77b\"/\"kube-root-ca.crt\"" Sep 11 14:08:26.879764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.879678 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-0bd77b\"/\"autores-model-0bd77b-kserve-self-signed-certs\"" Sep 11 14:08:26.887414 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.887393 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4"] Sep 11 14:08:26.905869 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.905843 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-home\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:26.905990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.905882 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc2cq\" (UniqueName: \"kubernetes.io/projected/720388c3-65f0-4eed-b38c-efbdcb8a09a0-kube-api-access-rc2cq\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:26.906067 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.906048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-model-cache\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:26.906145 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.906129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tmp-dir\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:26.906209 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.906157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-dshm\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:26.906209 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:26.906187 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tls-certs\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.006943 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.006914 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-model-cache\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.006951 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tmp-dir\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.006971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-dshm\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.006997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tls-certs\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.007044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-home\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007248 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.007069 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc2cq\" (UniqueName: \"kubernetes.io/projected/720388c3-65f0-4eed-b38c-efbdcb8a09a0-kube-api-access-rc2cq\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007390 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.007339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tmp-dir\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007514 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.007443 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-model-cache\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.007514 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.007488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-home\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.009438 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.009422 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-dshm\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.009539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.009522 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tls-certs\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.020321 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.020301 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc2cq\" (UniqueName: \"kubernetes.io/projected/720388c3-65f0-4eed-b38c-efbdcb8a09a0-kube-api-access-rc2cq\") pod \"autores-model-0bd77b-kserve-5649895dc5-b9wc4\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.181723 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.181645 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:27.186704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.186681 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:27.462018 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.461987 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4"] Sep 11 14:08:27.462889 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:08:27.462825 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod720388c3_65f0_4eed_b38c_efbdcb8a09a0.slice/crio-1c7fe6089451e5c553cc72b9f320103b85298cddb865186124a8a3d962ffb160 WatchSource:0}: Error finding container 1c7fe6089451e5c553cc72b9f320103b85298cddb865186124a8a3d962ffb160: Status 404 returned error can't find the container with id 1c7fe6089451e5c553cc72b9f320103b85298cddb865186124a8a3d962ffb160 Sep 11 14:08:27.872441 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.872407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" event={"ID":"720388c3-65f0-4eed-b38c-efbdcb8a09a0","Type":"ContainerStarted","Data":"da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569"} Sep 11 14:08:27.872441 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.872448 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" event={"ID":"720388c3-65f0-4eed-b38c-efbdcb8a09a0","Type":"ContainerStarted","Data":"1c7fe6089451e5c553cc72b9f320103b85298cddb865186124a8a3d962ffb160"} Sep 11 14:08:27.896048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:27.895995 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" podStartSLOduration=1.895977243 podStartE2EDuration="1.895977243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:27.893705479 +0000 UTC m=+924.763436742" watchObservedRunningTime="2026-09-11 14:08:27.895977243 +0000 UTC m=+924.765708543" Sep 11 14:08:37.186859 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.186825 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:37.187364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.186880 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:37.200177 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.200143 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:37.284194 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.284157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:37.907910 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.907878 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r"] Sep 11 14:08:37.908496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.908481 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b5cb804-7c0a-4896-b72a-1e0e2556b970" containerName="istio-proxy" Sep 11 14:08:37.912772 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.912751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:37.916320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.916299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:08:37.916416 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.916333 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:08:37.925746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.925725 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r"] Sep 11 14:08:37.952092 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:37.952061 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:08:38.017905 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.017868 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-383709f1-pcs7r\" (UID: \"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:38.119397 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.119364 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-383709f1-pcs7r\" (UID: \"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:38.119942 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.119923 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-383709f1-pcs7r\" (UID: \"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:38.224241 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.224155 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:38.380851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.380806 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:38.514388 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.514255 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r"] Sep 11 14:08:38.942170 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.942125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" event={"ID":"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9","Type":"ContainerStarted","Data":"f5ca31488ecbf4bc0289fbf6a40a3f2a93979f87155af8e2ce7652bc75ea5dce"} Sep 11 14:08:38.942344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.942226 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" event={"ID":"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9","Type":"ContainerStarted","Data":"bb3ffcf091e9f039ae773e9e92b3fa6da3e7f5bc5bff03d9aa5db520ea0ee658"} Sep 11 14:08:38.970073 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:38.970016 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" podStartSLOduration=1.970001519 podStartE2EDuration="1.970001519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:38.966229883 +0000 UTC m=+935.835961144" watchObservedRunningTime="2026-09-11 14:08:38.970001519 +0000 UTC m=+935.839732781" Sep 11 14:08:39.442636 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:39.442542 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:39.949624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:39.949589 2579 generic.go:356] "Generic (PLEG): container finished" podID="e0da8b05-655b-4a18-9b8e-b025bfb3a6c9" containerID="f5ca31488ecbf4bc0289fbf6a40a3f2a93979f87155af8e2ce7652bc75ea5dce" exitCode=0 Sep 11 14:08:39.949802 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:39.949638 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" event={"ID":"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9","Type":"ContainerDied","Data":"f5ca31488ecbf4bc0289fbf6a40a3f2a93979f87155af8e2ce7652bc75ea5dce"} Sep 11 14:08:40.336331 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:40.336293 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:41.097199 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.097176 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:41.148437 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.148408 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9-maas-api-service-ca\") pod \"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9\" (UID: \"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9\") " Sep 11 14:08:41.148828 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.148804 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0da8b05-655b-4a18-9b8e-b025bfb3a6c9" (UID: "e0da8b05-655b-4a18-9b8e-b025bfb3a6c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:08:41.249522 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.249440 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:41.374838 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.374804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:41.964042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.963960 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" Sep 11 14:08:41.964042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.964003 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r" event={"ID":"e0da8b05-655b-4a18-9b8e-b025bfb3a6c9","Type":"ContainerDied","Data":"bb3ffcf091e9f039ae773e9e92b3fa6da3e7f5bc5bff03d9aa5db520ea0ee658"} Sep 11 14:08:41.964309 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:41.964051 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb3ffcf091e9f039ae773e9e92b3fa6da3e7f5bc5bff03d9aa5db520ea0ee658" Sep 11 14:08:42.479090 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:42.479027 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:42.884790 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:42.884754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:43.864468 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:43.864436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:44.241549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:44.241455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:44.622614 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:44.622582 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:44.920432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:44.920348 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:45.023271 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:45.023241 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r"] Sep 11 14:08:45.026332 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:45.026301 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-383709f1-pcs7r"] Sep 11 14:08:45.221110 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:45.221012 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:45.643581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:45.643546 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0da8b05-655b-4a18-9b8e-b025bfb3a6c9" path="/var/lib/kubelet/pods/e0da8b05-655b-4a18-9b8e-b025bfb3a6c9/volumes" Sep 11 14:08:46.871855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:46.871825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:47.076759 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.076725 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn"] Sep 11 14:08:47.077200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.077187 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0da8b05-655b-4a18-9b8e-b025bfb3a6c9" containerName="revoke-keys" Sep 11 14:08:47.077249 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.077202 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0da8b05-655b-4a18-9b8e-b025bfb3a6c9" containerName="revoke-keys" Sep 11 14:08:47.077298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.077289 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0da8b05-655b-4a18-9b8e-b025bfb3a6c9" containerName="revoke-keys" Sep 11 14:08:47.079477 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.079462 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.082928 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.082911 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-61f2ffc9-openshift-default-dockercfg-6x8j7\"" Sep 11 14:08:47.115227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.115190 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn"] Sep 11 14:08:47.138635 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.138551 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:47.215646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215614 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzcn2\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-kube-api-access-nzcn2\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215665 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-podinfo\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215765 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-envoy\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215804 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istiod-ca-cert\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215857 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-socket\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215916 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-data\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215972 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-certs\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.215994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.215993 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-token\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.216186 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.216064 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-credential-socket\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.317194 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.317156 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-envoy\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.317454 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.317435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istiod-ca-cert\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.317642 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.317627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-socket\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.317762 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.317750 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-data\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.317893 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.317880 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-certs\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.317990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.317977 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-token\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.318138 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.318121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-credential-socket\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.318282 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.318267 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzcn2\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-kube-api-access-nzcn2\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.318408 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.318394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-podinfo\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.318571 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.318549 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-socket\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.318738 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.318295 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istiod-ca-cert\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.318852 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.318778 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-data\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.319068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.319049 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-credential-socket\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.319200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.319137 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-certs\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.320662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.320499 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-envoy\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.321721 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.321695 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-podinfo\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.349658 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.349625 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzcn2\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-kube-api-access-nzcn2\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.349845 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.349825 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-token\") pod \"e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.371078 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.371040 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:47.390017 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.389958 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:47.699881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.699857 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn"] Sep 11 14:08:47.701267 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:08:47.701240 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df9132b_ffcd_4059_9ade_9e0f763ef90c.slice/crio-cc1a5fde3cc19f6e81c54e8ec25706f431bd88aeb055686fac7d19afdab3dd23 WatchSource:0}: Error finding container cc1a5fde3cc19f6e81c54e8ec25706f431bd88aeb055686fac7d19afdab3dd23: Status 404 returned error can't find the container with id cc1a5fde3cc19f6e81c54e8ec25706f431bd88aeb055686fac7d19afdab3dd23 Sep 11 14:08:47.703501 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.703471 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:08:47.703601 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.703547 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:08:47.703601 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.703591 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:08:47.772243 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:47.772216 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:48.009542 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:48.009459 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" event={"ID":"7df9132b-ffcd-4059-9ade-9e0f763ef90c","Type":"ContainerStarted","Data":"4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130"} Sep 11 14:08:48.009542 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:48.009512 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" event={"ID":"7df9132b-ffcd-4059-9ade-9e0f763ef90c","Type":"ContainerStarted","Data":"cc1a5fde3cc19f6e81c54e8ec25706f431bd88aeb055686fac7d19afdab3dd23"} Sep 11 14:08:48.047183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:48.047092 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podStartSLOduration=1.047073414 podStartE2EDuration="1.047073414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:08:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:08:48.043382173 +0000 UTC m=+944.913113435" watchObservedRunningTime="2026-09-11 14:08:48.047073414 +0000 UTC m=+944.916804677" Sep 11 14:08:48.390563 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:48.390522 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:48.395646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:48.395621 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:48.876034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:48.876001 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:49.016415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:49.016388 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:49.017369 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:49.017346 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:08:49.473650 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:49.473609 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:49.622901 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:49.622860 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:50.147776 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:50.147739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:51.032304 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:51.032269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:51.335255 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:51.335224 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:51.491430 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:51.491395 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:51.736736 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:51.736652 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:52.262053 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:52.262017 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:52.371728 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:52.371694 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:52.748416 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:52.748385 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4"] Sep 11 14:08:52.763569 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:52.763530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:53.189760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.189729 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn"] Sep 11 14:08:53.193691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.193674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:53.197167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.197148 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:08:53.197278 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.197150 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:08:53.209225 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.209201 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn"] Sep 11 14:08:53.253621 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.253573 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:53.282916 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.282883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7326e-7604-40f2-8672-bfe63850b011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0bd77b-vzswn\" (UID: \"f4e7326e-7604-40f2-8672-bfe63850b011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:53.384782 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.384744 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7326e-7604-40f2-8672-bfe63850b011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0bd77b-vzswn\" (UID: \"f4e7326e-7604-40f2-8672-bfe63850b011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:53.385405 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.385385 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7326e-7604-40f2-8672-bfe63850b011-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-0bd77b-vzswn\" (UID: \"f4e7326e-7604-40f2-8672-bfe63850b011\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:53.504917 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.504842 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:53.784806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:53.784775 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn"] Sep 11 14:08:53.786093 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:08:53.786069 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4e7326e_7604_40f2_8672_bfe63850b011.slice/crio-ee0462e3376f35c00fb1ff0649d5e97e16909b43851cfee5beb69128228e3cb7 WatchSource:0}: Error finding container ee0462e3376f35c00fb1ff0649d5e97e16909b43851cfee5beb69128228e3cb7: Status 404 returned error can't find the container with id ee0462e3376f35c00fb1ff0649d5e97e16909b43851cfee5beb69128228e3cb7 Sep 11 14:08:54.047975 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:54.047939 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" event={"ID":"f4e7326e-7604-40f2-8672-bfe63850b011","Type":"ContainerStarted","Data":"ee0462e3376f35c00fb1ff0649d5e97e16909b43851cfee5beb69128228e3cb7"} Sep 11 14:08:54.259554 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:54.259511 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:55.056031 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:55.055995 2579 generic.go:356] "Generic (PLEG): container finished" podID="f4e7326e-7604-40f2-8672-bfe63850b011" containerID="f39b39652bf8faec4a92bd2a477234384d909b44c40b1cc50de276d996f9729c" exitCode=0 Sep 11 14:08:55.056453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:55.056069 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" event={"ID":"f4e7326e-7604-40f2-8672-bfe63850b011","Type":"ContainerDied","Data":"f39b39652bf8faec4a92bd2a477234384d909b44c40b1cc50de276d996f9729c"} Sep 11 14:08:55.375637 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:55.375555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:56.196406 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:56.196382 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:56.216377 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:56.216350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7326e-7604-40f2-8672-bfe63850b011-maas-api-service-ca\") pod \"f4e7326e-7604-40f2-8672-bfe63850b011\" (UID: \"f4e7326e-7604-40f2-8672-bfe63850b011\") " Sep 11 14:08:56.216952 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:56.216907 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4e7326e-7604-40f2-8672-bfe63850b011-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4e7326e-7604-40f2-8672-bfe63850b011" (UID: "f4e7326e-7604-40f2-8672-bfe63850b011"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:08:56.318289 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:56.318207 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4e7326e-7604-40f2-8672-bfe63850b011-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:08:56.352738 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:56.352702 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:57.069601 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:57.069574 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" Sep 11 14:08:57.069601 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:57.069591 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn" event={"ID":"f4e7326e-7604-40f2-8672-bfe63850b011","Type":"ContainerDied","Data":"ee0462e3376f35c00fb1ff0649d5e97e16909b43851cfee5beb69128228e3cb7"} Sep 11 14:08:57.069859 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:57.069634 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee0462e3376f35c00fb1ff0649d5e97e16909b43851cfee5beb69128228e3cb7" Sep 11 14:08:57.174660 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:57.174620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:08:58.002594 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:08:58.002532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:03.333668 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:03.333628 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:03.994894 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:03.994790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:04.094242 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.094209 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:04.194895 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.194865 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:04.230007 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.229984 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn"] Sep 11 14:09:04.232702 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.232683 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-0bd77b-vzswn"] Sep 11 14:09:04.298511 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.298477 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:04.609171 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.609081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:04.773224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:04.773194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:05.643826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:05.643788 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4e7326e-7604-40f2-8672-bfe63850b011" path="/var/lib/kubelet/pods/f4e7326e-7604-40f2-8672-bfe63850b011/volumes" Sep 11 14:09:05.661707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:05.661684 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:06.882299 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:06.882255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:07.768277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:07.768238 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" podUID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" containerName="main" containerID="cri-o://da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569" gracePeriod=45 Sep 11 14:09:07.935973 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:07.935932 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" podUID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused" Sep 11 14:09:08.053194 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.053165 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:09:08.138415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138384 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tls-certs\") pod \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " Sep 11 14:09:08.138607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-dshm\") pod \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " Sep 11 14:09:08.138607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-home\") pod \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " Sep 11 14:09:08.138607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138483 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-model-cache\") pod \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " Sep 11 14:09:08.138607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138526 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tmp-dir\") pod \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " Sep 11 14:09:08.138836 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138609 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc2cq\" (UniqueName: \"kubernetes.io/projected/720388c3-65f0-4eed-b38c-efbdcb8a09a0-kube-api-access-rc2cq\") pod \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\" (UID: \"720388c3-65f0-4eed-b38c-efbdcb8a09a0\") " Sep 11 14:09:08.138836 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138779 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "720388c3-65f0-4eed-b38c-efbdcb8a09a0" (UID: "720388c3-65f0-4eed-b38c-efbdcb8a09a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:08.138836 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138785 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-home" (OuterVolumeSpecName: "home") pod "720388c3-65f0-4eed-b38c-efbdcb8a09a0" (UID: "720388c3-65f0-4eed-b38c-efbdcb8a09a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:08.139003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138940 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "720388c3-65f0-4eed-b38c-efbdcb8a09a0" (UID: "720388c3-65f0-4eed-b38c-efbdcb8a09a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:08.139003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138965 2579 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-home\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:08.139003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.138982 2579 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-model-cache\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:08.141008 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.140981 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-dshm" (OuterVolumeSpecName: "dshm") pod "720388c3-65f0-4eed-b38c-efbdcb8a09a0" (UID: "720388c3-65f0-4eed-b38c-efbdcb8a09a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:08.141148 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.141032 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720388c3-65f0-4eed-b38c-efbdcb8a09a0-kube-api-access-rc2cq" (OuterVolumeSpecName: "kube-api-access-rc2cq") pod "720388c3-65f0-4eed-b38c-efbdcb8a09a0" (UID: "720388c3-65f0-4eed-b38c-efbdcb8a09a0"). InnerVolumeSpecName "kube-api-access-rc2cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:08.141148 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.141065 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "720388c3-65f0-4eed-b38c-efbdcb8a09a0" (UID: "720388c3-65f0-4eed-b38c-efbdcb8a09a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:09:08.146602 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.146579 2579 generic.go:356] "Generic (PLEG): container finished" podID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" containerID="da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569" exitCode=0 Sep 11 14:09:08.146688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.146610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" event={"ID":"720388c3-65f0-4eed-b38c-efbdcb8a09a0","Type":"ContainerDied","Data":"da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569"} Sep 11 14:09:08.146688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.146626 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" Sep 11 14:09:08.146688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.146636 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4" event={"ID":"720388c3-65f0-4eed-b38c-efbdcb8a09a0","Type":"ContainerDied","Data":"1c7fe6089451e5c553cc72b9f320103b85298cddb865186124a8a3d962ffb160"} Sep 11 14:09:08.146688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.146651 2579 scope.go:117] "RemoveContainer" containerID="da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569" Sep 11 14:09:08.163542 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.163528 2579 scope.go:117] "RemoveContainer" containerID="da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569" Sep 11 14:09:08.163779 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:09:08.163758 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569\": container with ID starting with da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569 not found: ID does not exist" containerID="da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569" Sep 11 14:09:08.163847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.163791 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569"} err="failed to get container status \"da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569\": rpc error: code = NotFound desc = could not find container \"da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569\": container with ID starting with da94bdac50cd27fd9bfbd1a6525c0d2d2ebba850b578316580e75b482f97c569 not found: ID does not exist" Sep 11 14:09:08.197248 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.197219 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4"] Sep 11 14:09:08.199831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.199807 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-0bd77b/autores-model-0bd77b-kserve-5649895dc5-b9wc4"] Sep 11 14:09:08.233081 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.233053 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:08.240476 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.240459 2579 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tls-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:08.240547 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.240478 2579 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-dshm\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:08.240547 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.240487 2579 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/720388c3-65f0-4eed-b38c-efbdcb8a09a0-tmp-dir\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:08.240547 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:08.240495 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc2cq\" (UniqueName: \"kubernetes.io/projected/720388c3-65f0-4eed-b38c-efbdcb8a09a0-kube-api-access-rc2cq\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:09.131438 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:09.131404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:09.643652 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:09.643619 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" path="/var/lib/kubelet/pods/720388c3-65f0-4eed-b38c-efbdcb8a09a0/volumes" Sep 11 14:09:10.111777 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:10.111743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:10.866656 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:10.866619 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:11.847368 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:11.847334 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:12.008732 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.008697 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:12.117192 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.117071 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:12.246860 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.246824 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f"] Sep 11 14:09:12.247190 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.247159 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" containerID="cri-o://4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e" gracePeriod=30 Sep 11 14:09:12.277378 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.277346 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:12.646751 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.646713 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:12.745675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:12.745641 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:13.009299 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:13.009210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:13.637301 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:13.637269 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:13.941241 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:13.941160 2579 patch_prober.go:28] interesting pod/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:09:13.941241 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:13.941212 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:09:14.196021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:14.195922 2579 generic.go:356] "Generic (PLEG): container finished" podID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerID="4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130" exitCode=0 Sep 11 14:09:14.196517 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:14.196035 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" event={"ID":"7df9132b-ffcd-4059-9ade-9e0f763ef90c","Type":"ContainerDied","Data":"4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130"} Sep 11 14:09:14.196678 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:14.196655 2579 scope.go:117] "RemoveContainer" containerID="4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130" Sep 11 14:09:14.198826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:14.198793 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:09:14.198973 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:14.198914 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:09:14.198973 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:14.198967 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:09:15.205875 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:15.205841 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" event={"ID":"7df9132b-ffcd-4059-9ade-9e0f763ef90c","Type":"ContainerStarted","Data":"581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53"} Sep 11 14:09:15.306819 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:15.306785 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:15.391213 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:15.391128 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:09:15.392483 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:15.392445 2579 patch_prober.go:28] interesting pod/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:09:15.392638 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:15.392509 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:09:16.122959 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:16.122918 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:16.223754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:16.223711 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:16.390455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:16.390361 2579 patch_prober.go:28] interesting pod/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:09:16.390761 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:16.390730 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:09:16.473163 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:16.473120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:17.393509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.393473 2579 patch_prober.go:28] interesting pod/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:09:17.393999 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.393537 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:09:17.551613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.551589 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:09:17.612119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.612074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:17.646165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.646136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-certs\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.647042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.646231 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ca79ba-0156-4783-a157-5bf4c53ee5db-istiod-ca-cert\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.647042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.646280 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hbdp\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-kube-api-access-4hbdp\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.647042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.646309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-credential-socket\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.647042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.646333 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-podinfo\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.647042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.646616 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-data\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.647951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.647885 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:17.648094 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.648074 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:17.648664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.648389 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ca79ba-0156-4783-a157-5bf4c53ee5db-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:09:17.649213 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.649190 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-data" (OuterVolumeSpecName: "istio-data") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:17.651509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.651487 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-envoy\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.653684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.652637 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-token\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.653684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.652755 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-socket\") pod \"58ca79ba-0156-4783-a157-5bf4c53ee5db\" (UID: \"58ca79ba-0156-4783-a157-5bf4c53ee5db\") " Sep 11 14:09:17.653684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.653397 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:17.653902 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.653874 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ca79ba-0156-4783-a157-5bf4c53ee5db-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.653972 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.653904 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.653972 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.653921 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.653972 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.653937 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.653972 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.653950 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.657073 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.657046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:09:17.657204 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.657086 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-kube-api-access-4hbdp" (OuterVolumeSpecName: "kube-api-access-4hbdp") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "kube-api-access-4hbdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:17.657588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.657567 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-token" (OuterVolumeSpecName: "istio-token") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:17.657674 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.657586 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "58ca79ba-0156-4783-a157-5bf4c53ee5db" (UID: "58ca79ba-0156-4783-a157-5bf4c53ee5db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:17.754707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.754676 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.754707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.754701 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.754707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.754710 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hbdp\" (UniqueName: \"kubernetes.io/projected/58ca79ba-0156-4783-a157-5bf4c53ee5db-kube-api-access-4hbdp\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:17.754942 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:17.754720 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ca79ba-0156-4783-a157-5bf4c53ee5db-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:18.227661 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.227629 2579 generic.go:356] "Generic (PLEG): container finished" podID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerID="4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e" exitCode=0 Sep 11 14:09:18.227833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.227677 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" Sep 11 14:09:18.227833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.227718 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" event={"ID":"58ca79ba-0156-4783-a157-5bf4c53ee5db","Type":"ContainerDied","Data":"4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e"} Sep 11 14:09:18.227833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.227775 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f" event={"ID":"58ca79ba-0156-4783-a157-5bf4c53ee5db","Type":"ContainerDied","Data":"a55fbc0ed03a25d4b4e9eb084c35c79e633cf6c1ed08a5a2367fef458599abfc"} Sep 11 14:09:18.227833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.227797 2579 scope.go:117] "RemoveContainer" containerID="4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e" Sep 11 14:09:18.238615 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.238596 2579 scope.go:117] "RemoveContainer" containerID="8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e" Sep 11 14:09:18.247185 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.247165 2579 scope.go:117] "RemoveContainer" containerID="4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e" Sep 11 14:09:18.247453 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:09:18.247436 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e\": container with ID starting with 4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e not found: ID does not exist" containerID="4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e" Sep 11 14:09:18.247496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.247465 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e"} err="failed to get container status \"4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e\": rpc error: code = NotFound desc = could not find container \"4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e\": container with ID starting with 4274a418517f08aa148a5059a2e2cde2cd7e34000da8815a974bd5228617f10e not found: ID does not exist" Sep 11 14:09:18.247496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.247481 2579 scope.go:117] "RemoveContainer" containerID="8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e" Sep 11 14:09:18.247719 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:09:18.247704 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e\": container with ID starting with 8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e not found: ID does not exist" containerID="8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e" Sep 11 14:09:18.247756 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.247723 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e"} err="failed to get container status \"8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e\": rpc error: code = NotFound desc = could not find container \"8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e\": container with ID starting with 8ca76fc36886ba5c8a66d45e6ad337c2fe701d1a40f003a141a3859dc835243e not found: ID does not exist" Sep 11 14:09:18.253594 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.253574 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f"] Sep 11 14:09:18.263293 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.263275 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-0bd77b-openshift-default-67c9df4dc8-tmg5f"] Sep 11 14:09:18.390548 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.390511 2579 patch_prober.go:28] interesting pod/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:09:18.390833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.390802 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:09:18.507321 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:18.507220 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:19.016853 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.016816 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:09:19.235028 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.234996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:19.336717 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.336680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:19.390951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.390899 2579 patch_prober.go:28] interesting pod/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Sep 11 14:09:19.391190 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.390980 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Sep 11 14:09:19.644148 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.644049 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" path="/var/lib/kubelet/pods/58ca79ba-0156-4783-a157-5bf4c53ee5db/volumes" Sep 11 14:09:19.651800 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.651768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:19.759056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.759022 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:19.852684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:19.852647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:20.154879 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.154842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:20.242643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.242598 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:20.338975 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.338939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:20.436829 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.436735 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:20.517287 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.517249 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn"] Sep 11 14:09:20.517472 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.517448 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" containerID="cri-o://581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53" gracePeriod=30 Sep 11 14:09:20.547132 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.547074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:20.680757 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:20.680726 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:09:21.040714 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:21.040680 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:21.244664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:21.244617 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:21.384838 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:21.384801 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:22.025388 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:22.025352 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:22.784624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:22.784591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:22.884926 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:22.884893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:23.828699 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:23.828663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:23.988666 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:23.988636 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:25.876383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.876362 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:09:25.962458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962381 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-certs\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962436 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istiod-ca-cert\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962460 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-socket\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962479 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-credential-socket\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962507 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-token\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962562 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzcn2\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-kube-api-access-nzcn2\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962669 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:25.962925 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962712 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-podinfo\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962925 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962796 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-envoy\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.962925 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962815 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:25.962925 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962824 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:25.962925 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-data\") pod \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\" (UID: \"7df9132b-ffcd-4059-9ade-9e0f763ef90c\") " Sep 11 14:09:25.963212 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.962944 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:09:25.963212 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.963050 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-data" (OuterVolumeSpecName: "istio-data") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:25.963330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.963309 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:25.963395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.963337 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:25.963395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.963352 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:25.963395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.963364 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:25.963395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.963377 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:25.964985 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.964963 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:09:25.965127 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.965020 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-token" (OuterVolumeSpecName: "istio-token") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:25.965418 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.965399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:09:25.965418 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:25.965403 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-kube-api-access-nzcn2" (OuterVolumeSpecName: "kube-api-access-nzcn2") pod "7df9132b-ffcd-4059-9ade-9e0f763ef90c" (UID: "7df9132b-ffcd-4059-9ade-9e0f763ef90c"). InnerVolumeSpecName "kube-api-access-nzcn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:09:26.064776 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.064748 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzcn2\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-kube-api-access-nzcn2\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:26.064776 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.064774 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:26.064918 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.064787 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:26.064918 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.064799 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7df9132b-ffcd-4059-9ade-9e0f763ef90c-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:09:26.295397 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.295367 2579 generic.go:356] "Generic (PLEG): container finished" podID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerID="581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53" exitCode=0 Sep 11 14:09:26.295553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.295420 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" Sep 11 14:09:26.295553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.295433 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" event={"ID":"7df9132b-ffcd-4059-9ade-9e0f763ef90c","Type":"ContainerDied","Data":"581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53"} Sep 11 14:09:26.295553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.295475 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn" event={"ID":"7df9132b-ffcd-4059-9ade-9e0f763ef90c","Type":"ContainerDied","Data":"cc1a5fde3cc19f6e81c54e8ec25706f431bd88aeb055686fac7d19afdab3dd23"} Sep 11 14:09:26.295553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.295496 2579 scope.go:117] "RemoveContainer" containerID="581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53" Sep 11 14:09:26.310953 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.310930 2579 scope.go:117] "RemoveContainer" containerID="4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130" Sep 11 14:09:26.320680 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.320647 2579 scope.go:117] "RemoveContainer" containerID="581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53" Sep 11 14:09:26.321060 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:09:26.321031 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53\": container with ID starting with 581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53 not found: ID does not exist" containerID="581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53" Sep 11 14:09:26.321148 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.321068 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53"} err="failed to get container status \"581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53\": rpc error: code = NotFound desc = could not find container \"581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53\": container with ID starting with 581e9d5dbbdb499cc654a30f34c3234c0a3c3122fca9e147ddd39c38177e0c53 not found: ID does not exist" Sep 11 14:09:26.321148 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.321084 2579 scope.go:117] "RemoveContainer" containerID="4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130" Sep 11 14:09:26.321362 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:09:26.321345 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130\": container with ID starting with 4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130 not found: ID does not exist" containerID="4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130" Sep 11 14:09:26.321408 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.321367 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130"} err="failed to get container status \"4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130\": rpc error: code = NotFound desc = could not find container \"4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130\": container with ID starting with 4dda5552c7b63a33b65b6e632e1c4a19f2784532ba8c18ce05dc50fac9502130 not found: ID does not exist" Sep 11 14:09:26.323590 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.323564 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn"] Sep 11 14:09:26.333885 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:26.333866 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-61f2ffc9-openshift-default-6fc59c5b6d-ckdbn"] Sep 11 14:09:27.643607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:27.643577 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" path="/var/lib/kubelet/pods/7df9132b-ffcd-4059-9ade-9e0f763ef90c/volumes" Sep 11 14:09:31.473509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:31.473475 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:31.590426 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:31.590357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:31.684346 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:31.684312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:31.861205 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:31.861173 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:32.252290 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:32.252197 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:41.721389 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:41.721361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:42.645701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:42.645667 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:43.407113 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:43.407075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:44.379748 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:44.379705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:46.099294 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:46.099262 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:47.574580 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:47.574546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:56.254650 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:56.254603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:09:58.124015 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:09:58.123985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:04.226416 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:04.226378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:04.338131 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:04.338080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:04.446515 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:04.446479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:12.065594 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.065559 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:12.130716 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.130676 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46"] Sep 11 14:10:12.131404 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131376 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" Sep 11 14:10:12.131404 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131404 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131423 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131432 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131460 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131467 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131479 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131487 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131506 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4e7326e-7604-40f2-8672-bfe63850b011" containerName="revoke-keys" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131515 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4e7326e-7604-40f2-8672-bfe63850b011" containerName="revoke-keys" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131532 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" containerName="main" Sep 11 14:10:12.131626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131540 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" containerName="main" Sep 11 14:10:12.132048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131648 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" Sep 11 14:10:12.132048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131663 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="720388c3-65f0-4eed-b38c-efbdcb8a09a0" containerName="main" Sep 11 14:10:12.132048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131670 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4e7326e-7604-40f2-8672-bfe63850b011" containerName="revoke-keys" Sep 11 14:10:12.132048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131681 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df9132b-ffcd-4059-9ade-9e0f763ef90c" containerName="istio-proxy" Sep 11 14:10:12.132048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131692 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" Sep 11 14:10:12.132048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.131703 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="58ca79ba-0156-4783-a157-5bf4c53ee5db" containerName="istio-proxy" Sep 11 14:10:12.134956 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.134935 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:12.140178 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.140154 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:10:12.140359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.140299 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:10:12.143975 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.143951 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46"] Sep 11 14:10:12.193400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.193368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f281e97-e220-476d-93d9-c70698db81ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46\" (UID: \"2f281e97-e220-476d-93d9-c70698db81ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:12.294400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.294366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f281e97-e220-476d-93d9-c70698db81ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46\" (UID: \"2f281e97-e220-476d-93d9-c70698db81ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:12.295219 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.295195 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f281e97-e220-476d-93d9-c70698db81ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46\" (UID: \"2f281e97-e220-476d-93d9-c70698db81ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:12.448844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.448785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:12.753038 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:12.753010 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46"] Sep 11 14:10:12.753516 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:10:12.753494 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f281e97_e220_476d_93d9_c70698db81ab.slice/crio-a2a984c967c77fc366aff3d4684ec61b5c5e3542b2f7fe37057d405ba8907d43 WatchSource:0}: Error finding container a2a984c967c77fc366aff3d4684ec61b5c5e3542b2f7fe37057d405ba8907d43: Status 404 returned error can't find the container with id a2a984c967c77fc366aff3d4684ec61b5c5e3542b2f7fe37057d405ba8907d43 Sep 11 14:10:13.058118 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:13.058068 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:13.601408 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:13.601376 2579 generic.go:356] "Generic (PLEG): container finished" podID="2f281e97-e220-476d-93d9-c70698db81ab" containerID="1530f62546b9c397f548f73b3d8f93456283dfedc5058202f65fb410a64746e0" exitCode=0 Sep 11 14:10:13.601751 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:13.601463 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" event={"ID":"2f281e97-e220-476d-93d9-c70698db81ab","Type":"ContainerDied","Data":"1530f62546b9c397f548f73b3d8f93456283dfedc5058202f65fb410a64746e0"} Sep 11 14:10:13.601751 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:13.601526 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" event={"ID":"2f281e97-e220-476d-93d9-c70698db81ab","Type":"ContainerStarted","Data":"a2a984c967c77fc366aff3d4684ec61b5c5e3542b2f7fe37057d405ba8907d43"} Sep 11 14:10:13.937083 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:13.937045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:14.639378 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:14.639336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:14.756618 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:14.756594 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:14.822956 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:14.822926 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f281e97-e220-476d-93d9-c70698db81ab-maas-api-service-ca\") pod \"2f281e97-e220-476d-93d9-c70698db81ab\" (UID: \"2f281e97-e220-476d-93d9-c70698db81ab\") " Sep 11 14:10:14.823287 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:14.823265 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f281e97-e220-476d-93d9-c70698db81ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f281e97-e220-476d-93d9-c70698db81ab" (UID: "2f281e97-e220-476d-93d9-c70698db81ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:10:14.923901 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:14.923823 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f281e97-e220-476d-93d9-c70698db81ab-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:10:15.606543 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:15.606502 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:15.617081 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:15.617055 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" Sep 11 14:10:15.617237 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:15.617053 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46" event={"ID":"2f281e97-e220-476d-93d9-c70698db81ab","Type":"ContainerDied","Data":"a2a984c967c77fc366aff3d4684ec61b5c5e3542b2f7fe37057d405ba8907d43"} Sep 11 14:10:15.617237 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:15.617194 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2a984c967c77fc366aff3d4684ec61b5c5e3542b2f7fe37057d405ba8907d43" Sep 11 14:10:20.360402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:20.360360 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:22.738705 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:22.738666 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:22.845803 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:22.845765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:22.937640 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:22.937606 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:23.002031 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:23.001962 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46"] Sep 11 14:10:23.009040 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:23.009012 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-25f5d3-p5v46"] Sep 11 14:10:23.029423 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:23.029389 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:23.338770 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:23.338737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:23.467807 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:23.467770 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:23.643046 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:23.642973 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f281e97-e220-476d-93d9-c70698db81ab" path="/var/lib/kubelet/pods/2f281e97-e220-476d-93d9-c70698db81ab/volumes" Sep 11 14:10:24.574996 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:24.574969 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:25.368275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:25.368245 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:27.582573 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:27.582538 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:27.709060 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:27.709029 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:27.793891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:27.793857 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:30.139546 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:30.139508 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:30.236036 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:30.235996 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:30.510721 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:30.510622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:30.664288 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:30.664234 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:30.916169 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:30.916134 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:31.242570 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.242495 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc"] Sep 11 14:10:31.242966 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.242952 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f281e97-e220-476d-93d9-c70698db81ab" containerName="revoke-keys" Sep 11 14:10:31.243024 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.242967 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f281e97-e220-476d-93d9-c70698db81ab" containerName="revoke-keys" Sep 11 14:10:31.243060 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.243028 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f281e97-e220-476d-93d9-c70698db81ab" containerName="revoke-keys" Sep 11 14:10:31.246021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.246004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.251163 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.251140 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-f83766-openshift-default-dockercfg-f56v8\"" Sep 11 14:10:31.261433 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.261412 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc"] Sep 11 14:10:31.268176 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.268149 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:31.369851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.369822 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-envoy\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.369856 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-podinfo\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.369881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-socket\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.369976 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-certs\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370171 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.370032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-credential-socket\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370171 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.370067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgmg2\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-kube-api-access-zgmg2\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370171 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.370085 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4a0372-dc79-4590-827c-7bf174cf50ea-istiod-ca-cert\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370268 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.370189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-data\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.370268 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.370217 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-token\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.466727 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.466681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:31.470872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.470845 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-credential-socket\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471014 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.470890 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgmg2\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-kube-api-access-zgmg2\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471079 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471005 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4a0372-dc79-4590-827c-7bf174cf50ea-istiod-ca-cert\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471127 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-data\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-token\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471273 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-envoy\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471273 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-podinfo\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471379 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471271 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-credential-socket\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471379 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-socket\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471379 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471367 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-certs\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471529 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471460 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-data\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471550 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-socket\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471662 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-certs\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.471823 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.471805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4a0372-dc79-4590-827c-7bf174cf50ea-istiod-ca-cert\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.473693 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.473671 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-envoy\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.473890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.473873 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-podinfo\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.495565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.495504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-token\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.506733 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.506701 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgmg2\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-kube-api-access-zgmg2\") pod \"e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.556172 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.556150 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:31.847855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.847814 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc"] Sep 11 14:10:31.847995 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:10:31.847913 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a4a0372_dc79_4590_827c_7bf174cf50ea.slice/crio-5efc6a2cab2a945f9070966065f8c584ec1bb6c66482c8a25f62e0179d9e2190 WatchSource:0}: Error finding container 5efc6a2cab2a945f9070966065f8c584ec1bb6c66482c8a25f62e0179d9e2190: Status 404 returned error can't find the container with id 5efc6a2cab2a945f9070966065f8c584ec1bb6c66482c8a25f62e0179d9e2190 Sep 11 14:10:31.850293 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.850264 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:10:31.850390 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.850321 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:10:31.850390 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:31.850347 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:10:32.727975 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:32.727940 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" event={"ID":"1a4a0372-dc79-4590-827c-7bf174cf50ea","Type":"ContainerStarted","Data":"32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102"} Sep 11 14:10:32.728431 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:32.727995 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" event={"ID":"1a4a0372-dc79-4590-827c-7bf174cf50ea","Type":"ContainerStarted","Data":"5efc6a2cab2a945f9070966065f8c584ec1bb6c66482c8a25f62e0179d9e2190"} Sep 11 14:10:32.761528 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:32.761477 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" podStartSLOduration=1.761460686 podStartE2EDuration="1.761460686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:10:32.756526464 +0000 UTC m=+1049.626257752" watchObservedRunningTime="2026-09-11 14:10:32.761460686 +0000 UTC m=+1049.631191947" Sep 11 14:10:33.112096 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.112056 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:33.214444 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.214382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:33.317231 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.317190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:33.556610 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.556575 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:33.561496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.561474 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:33.734512 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.734478 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:33.735387 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:33.735366 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:10:38.254355 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:38.254321 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:38.984334 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:38.984288 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:45.264852 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:45.264821 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:45.694959 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:45.694925 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:46.108822 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:46.108791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:46.559488 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:46.559457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:46.676324 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:46.676294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:47.236606 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:47.236570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:48.434424 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:48.434364 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:48.910468 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:48.910424 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:55.796745 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:55.796662 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:56.021632 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:56.021599 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:56.138060 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:56.137982 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:56.238686 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:56.238657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:56.941096 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:56.941050 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:57.009589 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:57.009561 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc"] Sep 11 14:10:57.009795 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:57.009774 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" podUID="1a4a0372-dc79-4590-827c-7bf174cf50ea" containerName="istio-proxy" containerID="cri-o://32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102" gracePeriod=30 Sep 11 14:10:57.168792 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:57.168757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:57.377155 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:57.377092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:57.487156 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:57.487097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:57.957062 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:57.957032 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:10:58.926640 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:10:58.926604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:00.085795 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:00.085758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:02.261149 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.261127 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:11:02.353893 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.353867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-data\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.353927 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-podinfo\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.353985 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-socket\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354007 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-token\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354029 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgmg2\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-kube-api-access-zgmg2\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354306 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354071 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4a0372-dc79-4590-827c-7bf174cf50ea-istiod-ca-cert\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354306 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354139 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-envoy\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354306 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354135 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:02.354306 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354181 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-certs\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354306 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-credential-socket\") pod \"1a4a0372-dc79-4590-827c-7bf174cf50ea\" (UID: \"1a4a0372-dc79-4590-827c-7bf174cf50ea\") " Sep 11 14:11:02.354574 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354338 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:02.354574 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354555 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:02.354681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354566 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a4a0372-dc79-4590-827c-7bf174cf50ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:11:02.354681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354636 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.354681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354657 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.354681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.354671 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:02.356431 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.356402 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:02.356431 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.356425 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:11:02.356588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.356470 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:02.356588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.356504 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-kube-api-access-zgmg2" (OuterVolumeSpecName: "kube-api-access-zgmg2") pod "1a4a0372-dc79-4590-827c-7bf174cf50ea" (UID: "1a4a0372-dc79-4590-827c-7bf174cf50ea"). InnerVolumeSpecName "kube-api-access-zgmg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:02.386709 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.386687 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:02.455914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455891 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.455914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455911 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgmg2\" (UniqueName: \"kubernetes.io/projected/1a4a0372-dc79-4590-827c-7bf174cf50ea-kube-api-access-zgmg2\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.456049 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455920 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a4a0372-dc79-4590-827c-7bf174cf50ea-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.456049 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455929 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.456049 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455937 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.456049 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455944 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a4a0372-dc79-4590-827c-7bf174cf50ea-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.456049 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.455951 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a4a0372-dc79-4590-827c-7bf174cf50ea-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:02.651059 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.650947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:02.928565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.928466 2579 generic.go:356] "Generic (PLEG): container finished" podID="1a4a0372-dc79-4590-827c-7bf174cf50ea" containerID="32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102" exitCode=0 Sep 11 14:11:02.928565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.928516 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" Sep 11 14:11:02.928767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.928560 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" event={"ID":"1a4a0372-dc79-4590-827c-7bf174cf50ea","Type":"ContainerDied","Data":"32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102"} Sep 11 14:11:02.928767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.928614 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc" event={"ID":"1a4a0372-dc79-4590-827c-7bf174cf50ea","Type":"ContainerDied","Data":"5efc6a2cab2a945f9070966065f8c584ec1bb6c66482c8a25f62e0179d9e2190"} Sep 11 14:11:02.928767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.928643 2579 scope.go:117] "RemoveContainer" containerID="32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102" Sep 11 14:11:02.939644 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.939626 2579 scope.go:117] "RemoveContainer" containerID="32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102" Sep 11 14:11:02.939919 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:11:02.939897 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102\": container with ID starting with 32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102 not found: ID does not exist" containerID="32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102" Sep 11 14:11:02.939990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.939932 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102"} err="failed to get container status \"32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102\": rpc error: code = NotFound desc = could not find container \"32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102\": container with ID starting with 32c6f1213d79edf1b3330169720a2083fb4623175220d8ebee4d2f378e457102 not found: ID does not exist" Sep 11 14:11:02.958586 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.958563 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc"] Sep 11 14:11:02.963883 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:02.963863 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-f83766-openshift-default-67b9ff58fb-5swhc"] Sep 11 14:11:03.410929 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.410899 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:03.565087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.565058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:03.644813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.644786 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a4a0372-dc79-4590-827c-7bf174cf50ea" path="/var/lib/kubelet/pods/1a4a0372-dc79-4590-827c-7bf174cf50ea/volumes" Sep 11 14:11:03.749627 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.749548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:03.976962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.976931 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8"] Sep 11 14:11:03.977396 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.977384 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a4a0372-dc79-4590-827c-7bf174cf50ea" containerName="istio-proxy" Sep 11 14:11:03.977447 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.977398 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a4a0372-dc79-4590-827c-7bf174cf50ea" containerName="istio-proxy" Sep 11 14:11:03.977483 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.977461 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a4a0372-dc79-4590-827c-7bf174cf50ea" containerName="istio-proxy" Sep 11 14:11:03.981915 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.981897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:03.985807 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:03.985787 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3e88320d-openshift-default-dockercfg-lfllr\"" Sep 11 14:11:04.025824 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.025770 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8"] Sep 11 14:11:04.063007 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.062974 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:04.068151 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-envoy\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068319 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068171 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faeac48-25e6-4573-bffb-c35cd005ee1e-istiod-ca-cert\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068319 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068202 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-certs\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068319 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-data\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068319 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068255 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-credential-socket\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068567 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068350 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vh77\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-kube-api-access-2vh77\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068567 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068434 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-socket\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068567 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068477 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-token\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.068735 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.068598 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-podinfo\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169247 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-credential-socket\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169298 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vh77\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-kube-api-access-2vh77\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169335 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-socket\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-token\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169464 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-podinfo\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169507 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-envoy\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faeac48-25e6-4573-bffb-c35cd005ee1e-istiod-ca-cert\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-certs\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169621 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-data\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169908 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169698 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-credential-socket\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.169908 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169710 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-socket\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.170010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169944 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-data\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.170010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.169985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-certs\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.170322 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.170291 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faeac48-25e6-4573-bffb-c35cd005ee1e-istiod-ca-cert\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.172009 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.171974 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-envoy\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.172129 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.172046 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-podinfo\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.184742 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.184721 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-token\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.185609 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.185588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vh77\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-kube-api-access-2vh77\") pod \"e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.292763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.292704 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:04.491526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.491485 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:04.582044 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.582021 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8"] Sep 11 14:11:04.582610 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:11:04.582586 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0faeac48_25e6_4573_bffb_c35cd005ee1e.slice/crio-b4187768cf5d560596439a8680aa3270ee84dddc8069eef70dde80e5ea414846 WatchSource:0}: Error finding container b4187768cf5d560596439a8680aa3270ee84dddc8069eef70dde80e5ea414846: Status 404 returned error can't find the container with id b4187768cf5d560596439a8680aa3270ee84dddc8069eef70dde80e5ea414846 Sep 11 14:11:04.585002 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.584975 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:04.585069 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.585032 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:04.585121 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.585067 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:04.944957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.944857 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" event={"ID":"0faeac48-25e6-4573-bffb-c35cd005ee1e","Type":"ContainerStarted","Data":"10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c"} Sep 11 14:11:04.944957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.944911 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" event={"ID":"0faeac48-25e6-4573-bffb-c35cd005ee1e","Type":"ContainerStarted","Data":"b4187768cf5d560596439a8680aa3270ee84dddc8069eef70dde80e5ea414846"} Sep 11 14:11:04.970788 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:04.970739 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podStartSLOduration=1.9707230070000001 podStartE2EDuration="1.970723007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:04.968059405 +0000 UTC m=+1081.837790668" watchObservedRunningTime="2026-09-11 14:11:04.970723007 +0000 UTC m=+1081.840454269" Sep 11 14:11:05.293021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:05.292941 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:05.297963 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:05.297941 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:05.951398 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:05.951367 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:05.952335 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:05.952316 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:06.709751 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:06.709717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:06.839624 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:06.839587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:07.060701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:07.060664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:07.189242 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:07.189159 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:08.693701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:08.693664 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:09.754906 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:09.754871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:09.862149 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:09.862119 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm"] Sep 11 14:11:09.862351 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:09.862327 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" podUID="f12c7e3c-a5dc-4526-9044-f937e546d114" containerName="payload-processing" containerID="cri-o://f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7" gracePeriod=30 Sep 11 14:11:09.913674 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:09.913642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:10.119869 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.119847 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:11:10.142601 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.142567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:10.230499 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.230472 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpkpm\" (UniqueName: \"kubernetes.io/projected/f12c7e3c-a5dc-4526-9044-f937e546d114-kube-api-access-bpkpm\") pod \"f12c7e3c-a5dc-4526-9044-f937e546d114\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " Sep 11 14:11:10.230656 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.230523 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c7e3c-a5dc-4526-9044-f937e546d114-plugins-config-volume\") pod \"f12c7e3c-a5dc-4526-9044-f937e546d114\" (UID: \"f12c7e3c-a5dc-4526-9044-f937e546d114\") " Sep 11 14:11:10.230937 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.230915 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12c7e3c-a5dc-4526-9044-f937e546d114-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f12c7e3c-a5dc-4526-9044-f937e546d114" (UID: "f12c7e3c-a5dc-4526-9044-f937e546d114"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:11:10.232740 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.232720 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12c7e3c-a5dc-4526-9044-f937e546d114-kube-api-access-bpkpm" (OuterVolumeSpecName: "kube-api-access-bpkpm") pod "f12c7e3c-a5dc-4526-9044-f937e546d114" (UID: "f12c7e3c-a5dc-4526-9044-f937e546d114"). InnerVolumeSpecName "kube-api-access-bpkpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:10.331977 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.331953 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpkpm\" (UniqueName: \"kubernetes.io/projected/f12c7e3c-a5dc-4526-9044-f937e546d114-kube-api-access-bpkpm\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:10.332133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.331986 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12c7e3c-a5dc-4526-9044-f937e546d114-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:10.474539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.474500 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:10.785657 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.785570 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:10.946844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.946805 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:10.982583 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.982559 2579 generic.go:356] "Generic (PLEG): container finished" podID="f12c7e3c-a5dc-4526-9044-f937e546d114" containerID="f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7" exitCode=0 Sep 11 14:11:10.982701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.982615 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" Sep 11 14:11:10.982701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.982646 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" event={"ID":"f12c7e3c-a5dc-4526-9044-f937e546d114","Type":"ContainerDied","Data":"f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7"} Sep 11 14:11:10.982781 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.982726 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" event={"ID":"f12c7e3c-a5dc-4526-9044-f937e546d114","Type":"ContainerDied","Data":"fd0f3d92d34def5d64db4923c3f551e4ac268b8a805596f32cb1262328d4708d"} Sep 11 14:11:10.982781 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.982755 2579 scope.go:117] "RemoveContainer" containerID="f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7" Sep 11 14:11:10.992959 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.992945 2579 scope.go:117] "RemoveContainer" containerID="f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7" Sep 11 14:11:10.993233 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:11:10.993211 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7\": container with ID starting with f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7 not found: ID does not exist" containerID="f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7" Sep 11 14:11:10.993299 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:10.993241 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7"} err="failed to get container status \"f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7\": rpc error: code = NotFound desc = could not find container \"f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7\": container with ID starting with f0665f4ceb929579dfda4fda5fd36a4e732b95eabe0c1ab7dba128cff025a5d7 not found: ID does not exist" Sep 11 14:11:11.013616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.013597 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm"] Sep 11 14:11:11.022491 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.022467 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm"] Sep 11 14:11:11.061347 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.061312 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:11.092196 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.092165 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-abfca2-5c7b444475-bg8fm" podUID="f12c7e3c-a5dc-4526-9044-f937e546d114" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.53:9004: i/o timeout" Sep 11 14:11:11.577794 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.577761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:11.643641 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.643606 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f12c7e3c-a5dc-4526-9044-f937e546d114" path="/var/lib/kubelet/pods/f12c7e3c-a5dc-4526-9044-f937e546d114/volumes" Sep 11 14:11:11.750195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:11.750169 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:12.270868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:12.270837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:13.261477 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:13.261448 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:14.053207 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:14.053171 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:17.870119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:17.870075 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:17.977823 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:17.977791 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:18.148736 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:18.148661 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9"] Sep 11 14:11:18.148897 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:18.148872 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" containerID="cri-o://33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3" gracePeriod=30 Sep 11 14:11:18.190234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:18.190200 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:18.893641 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:18.893588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:19.962346 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:19.962304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:21.041140 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:21.041094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:21.978127 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:21.973563 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:22.891288 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:22.891252 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:23.444487 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.444463 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:11:23.461269 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461238 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-certs\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461311 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-socket\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461345 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6add6870-c5dd-426f-bf7e-07d762ae299f-istiod-ca-cert\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461364 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-credential-socket\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461421 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwv5q\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-kube-api-access-pwv5q\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461440 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-data\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461462 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-envoy\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461499 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-token\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461519 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:23.461566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461555 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-podinfo\") pod \"6add6870-c5dd-426f-bf7e-07d762ae299f\" (UID: \"6add6870-c5dd-426f-bf7e-07d762ae299f\") " Sep 11 14:11:23.461865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:23.461865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461750 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-data" (OuterVolumeSpecName: "istio-data") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:23.461865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461757 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6add6870-c5dd-426f-bf7e-07d762ae299f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:11:23.462050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461933 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.462050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461952 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.462050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461966 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.462050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.461981 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6add6870-c5dd-426f-bf7e-07d762ae299f-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.462050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.462011 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:23.463788 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.463763 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-kube-api-access-pwv5q" (OuterVolumeSpecName: "kube-api-access-pwv5q") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "kube-api-access-pwv5q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:23.463951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.463929 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-token" (OuterVolumeSpecName: "istio-token") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:23.464321 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.464300 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:23.464564 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.464543 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6add6870-c5dd-426f-bf7e-07d762ae299f" (UID: "6add6870-c5dd-426f-bf7e-07d762ae299f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:11:23.563191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.563155 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.563191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.563190 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwv5q\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-kube-api-access-pwv5q\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.563191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.563200 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.563393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.563208 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.563393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.563215 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6add6870-c5dd-426f-bf7e-07d762ae299f-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:23.735822 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:23.735727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:24.068684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.068645 2579 generic.go:356] "Generic (PLEG): container finished" podID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerID="33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3" exitCode=0 Sep 11 14:11:24.069183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.068820 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" event={"ID":"6add6870-c5dd-426f-bf7e-07d762ae299f","Type":"ContainerDied","Data":"33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3"} Sep 11 14:11:24.069183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.068870 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" event={"ID":"6add6870-c5dd-426f-bf7e-07d762ae299f","Type":"ContainerDied","Data":"4628420af3ca78bfb0f7b43919b3809e2488ea9a1eecd8a267069b48c3196844"} Sep 11 14:11:24.069183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.068902 2579 scope.go:117] "RemoveContainer" containerID="33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3" Sep 11 14:11:24.069183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.069073 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9" Sep 11 14:11:24.096279 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.090648 2579 scope.go:117] "RemoveContainer" containerID="06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1" Sep 11 14:11:24.099746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.099714 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9"] Sep 11 14:11:24.102627 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.102603 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-abfca2-openshift-default-5899596d56-d48s9"] Sep 11 14:11:24.107003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.106984 2579 scope.go:117] "RemoveContainer" containerID="33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3" Sep 11 14:11:24.107305 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:11:24.107285 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3\": container with ID starting with 33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3 not found: ID does not exist" containerID="33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3" Sep 11 14:11:24.107365 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.107315 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3"} err="failed to get container status \"33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3\": rpc error: code = NotFound desc = could not find container \"33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3\": container with ID starting with 33670fe8f4efff81a8ba770a8ce65c8c9c36319f5e9be4d2aa442ed72976c0e3 not found: ID does not exist" Sep 11 14:11:24.107365 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.107333 2579 scope.go:117] "RemoveContainer" containerID="06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1" Sep 11 14:11:24.107594 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:11:24.107567 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1\": container with ID starting with 06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1 not found: ID does not exist" containerID="06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1" Sep 11 14:11:24.107635 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.107602 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1"} err="failed to get container status \"06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1\": rpc error: code = NotFound desc = could not find container \"06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1\": container with ID starting with 06a5d5f78537bc5bc262b3642b24d722986279ea72a5a7434a573ee9312c80f1 not found: ID does not exist" Sep 11 14:11:24.642951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:24.642921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:25.645455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:25.645422 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" path="/var/lib/kubelet/pods/6add6870-c5dd-426f-bf7e-07d762ae299f/volumes" Sep 11 14:11:26.739303 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:26.739266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:28.095074 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:28.095045 2579 generic.go:356] "Generic (PLEG): container finished" podID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerID="10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c" exitCode=0 Sep 11 14:11:28.095493 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:28.095135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" event={"ID":"0faeac48-25e6-4573-bffb-c35cd005ee1e","Type":"ContainerDied","Data":"10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c"} Sep 11 14:11:28.095541 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:28.095514 2579 scope.go:117] "RemoveContainer" containerID="10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c" Sep 11 14:11:28.096574 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:28.096528 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:28.096638 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:28.096623 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:28.096674 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:28.096662 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:29.102871 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:29.102837 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" event={"ID":"0faeac48-25e6-4573-bffb-c35cd005ee1e","Type":"ContainerStarted","Data":"43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208"} Sep 11 14:11:29.293791 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:29.293743 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:29.294010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:29.293956 2579 patch_prober.go:28] interesting pod/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:11:29.294144 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:29.294001 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 14:11:29.793686 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:29.793642 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:30.293820 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:30.293782 2579 patch_prober.go:28] interesting pod/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:11:30.294241 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:30.293839 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 14:11:31.293281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:31.293246 2579 patch_prober.go:28] interesting pod/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:11:31.293452 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:31.293312 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 14:11:32.249539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:32.249506 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:32.293878 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:32.293839 2579 patch_prober.go:28] interesting pod/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:11:32.294046 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:32.293899 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 14:11:33.004323 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.004291 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:33.293196 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.293082 2579 patch_prober.go:28] interesting pod/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Sep 11 14:11:33.293630 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.293178 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Sep 11 14:11:33.749137 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.749083 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:33.902704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.902656 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl"] Sep 11 14:11:33.903281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.903235 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" podUID="db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" containerName="payload-pre-processing" containerID="cri-o://60a6ad5a23c97618344888fd5c8508ebfe472a73b4df8333045710808c3d4816" gracePeriod=30 Sep 11 14:11:33.930940 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:33.930908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:34.087962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.087889 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:34.138725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.138691 2579 generic.go:356] "Generic (PLEG): container finished" podID="db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" containerID="60a6ad5a23c97618344888fd5c8508ebfe472a73b4df8333045710808c3d4816" exitCode=0 Sep 11 14:11:34.138904 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.138763 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" event={"ID":"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd","Type":"ContainerDied","Data":"60a6ad5a23c97618344888fd5c8508ebfe472a73b4df8333045710808c3d4816"} Sep 11 14:11:34.173133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.173092 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:11:34.181234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.181202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:34.261302 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.261271 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf9nq\" (UniqueName: \"kubernetes.io/projected/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-kube-api-access-cf9nq\") pod \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " Sep 11 14:11:34.261464 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.261370 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-plugins-config-volume\") pod \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\" (UID: \"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd\") " Sep 11 14:11:34.261754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.261732 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" (UID: "db2b5b62-cfe5-4e6d-b0bf-84d2033988cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:11:34.263560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.263533 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-kube-api-access-cf9nq" (OuterVolumeSpecName: "kube-api-access-cf9nq") pod "db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" (UID: "db2b5b62-cfe5-4e6d-b0bf-84d2033988cd"). InnerVolumeSpecName "kube-api-access-cf9nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:34.297588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.297562 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:34.297965 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.297894 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:34.298677 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.298656 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:11:34.362651 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.362613 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf9nq\" (UniqueName: \"kubernetes.io/projected/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-kube-api-access-cf9nq\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:34.362651 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.362646 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:34.549531 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:34.549501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:35.133320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.133294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:35.146828 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.146803 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" Sep 11 14:11:35.146960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.146846 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl" event={"ID":"db2b5b62-cfe5-4e6d-b0bf-84d2033988cd","Type":"ContainerDied","Data":"6bc829550f20a11383da6873e05bd546703c0f0f47f74327d556cb3baf7a5371"} Sep 11 14:11:35.146960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.146900 2579 scope.go:117] "RemoveContainer" containerID="60a6ad5a23c97618344888fd5c8508ebfe472a73b4df8333045710808c3d4816" Sep 11 14:11:35.175351 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.175327 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl"] Sep 11 14:11:35.178554 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.178535 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-80fb45-6b59fcb6f8-mprgl"] Sep 11 14:11:35.643077 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:35.643044 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" path="/var/lib/kubelet/pods/db2b5b62-cfe5-4e6d-b0bf-84d2033988cd/volumes" Sep 11 14:11:36.197952 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:36.197920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:36.297033 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:36.296995 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:36.886911 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:36.886876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:37.930767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:37.930728 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:38.829270 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:38.829241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:40.491899 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:40.491868 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:40.884494 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:40.884461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:40.981835 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:40.981804 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:41.140556 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:41.140468 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n"] Sep 11 14:11:41.140732 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:41.140694 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" containerID="cri-o://698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24" gracePeriod=30 Sep 11 14:11:41.164252 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:41.164215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:41.706291 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:41.706259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:42.066692 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.066666 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl"] Sep 11 14:11:42.067090 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067078 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" Sep 11 14:11:42.067157 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067092 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" Sep 11 14:11:42.067157 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067119 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f12c7e3c-a5dc-4526-9044-f937e546d114" containerName="payload-processing" Sep 11 14:11:42.067157 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067130 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f12c7e3c-a5dc-4526-9044-f937e546d114" containerName="payload-processing" Sep 11 14:11:42.067258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067156 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" Sep 11 14:11:42.067258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067166 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" Sep 11 14:11:42.067258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067179 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" containerName="payload-pre-processing" Sep 11 14:11:42.067258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067186 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" containerName="payload-pre-processing" Sep 11 14:11:42.067391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067265 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" Sep 11 14:11:42.067391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067274 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="db2b5b62-cfe5-4e6d-b0bf-84d2033988cd" containerName="payload-pre-processing" Sep 11 14:11:42.067391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067282 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6add6870-c5dd-426f-bf7e-07d762ae299f" containerName="istio-proxy" Sep 11 14:11:42.067391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.067289 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f12c7e3c-a5dc-4526-9044-f937e546d114" containerName="payload-processing" Sep 11 14:11:42.070245 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.070229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.073853 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.073836 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-fb65d4-openshift-default-dockercfg-6fgwx\"" Sep 11 14:11:42.093891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.093865 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl"] Sep 11 14:11:42.131633 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.131605 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:42.134583 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134557 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-certs\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134750 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134610 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-data\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134750 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134640 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flkkx\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-kube-api-access-flkkx\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134750 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-podinfo\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-credential-socket\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134805 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637c29a6-56e5-4dc2-b841-78d65be5961f-istiod-ca-cert\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-envoy\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.134872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134867 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-socket\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.135001 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.134883 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-token\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.226157 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.226089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:42.235471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-credential-socket\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235484 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637c29a6-56e5-4dc2-b841-78d65be5961f-istiod-ca-cert\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235699 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235617 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-envoy\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235722 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-socket\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235754 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-token\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235834 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-certs\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235862 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-credential-socket\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.235982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235952 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-data\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.236039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.235979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flkkx\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-kube-api-access-flkkx\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.236039 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.236026 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-podinfo\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.236176 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.236038 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-socket\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.236176 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.236117 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-certs\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.236287 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.236253 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637c29a6-56e5-4dc2-b841-78d65be5961f-istiod-ca-cert\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.236287 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.236268 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-data\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.238131 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.238086 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-envoy\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.238451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.238434 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-podinfo\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.245170 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.245143 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flkkx\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-kube-api-access-flkkx\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.248397 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.248377 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-token\") pod \"e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.380573 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.380500 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:42.682591 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.682502 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl"] Sep 11 14:11:42.684810 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:11:42.684779 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod637c29a6_56e5_4dc2_b841_78d65be5961f.slice/crio-a5d21d910d14a2250235f224bc91243a4ae904b7a055f337db195268ad391862 WatchSource:0}: Error finding container a5d21d910d14a2250235f224bc91243a4ae904b7a055f337db195268ad391862: Status 404 returned error can't find the container with id a5d21d910d14a2250235f224bc91243a4ae904b7a055f337db195268ad391862 Sep 11 14:11:42.687088 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.687059 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:42.687186 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.687143 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:42.687186 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:42.687170 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:11:43.206291 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:43.206252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" event={"ID":"637c29a6-56e5-4dc2-b841-78d65be5961f","Type":"ContainerStarted","Data":"55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda"} Sep 11 14:11:43.206737 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:43.206305 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" event={"ID":"637c29a6-56e5-4dc2-b841-78d65be5961f","Type":"ContainerStarted","Data":"a5d21d910d14a2250235f224bc91243a4ae904b7a055f337db195268ad391862"} Sep 11 14:11:43.231387 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:43.231336 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podStartSLOduration=1.231315139 podStartE2EDuration="1.231315139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:11:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:11:43.230285477 +0000 UTC m=+1120.100016751" watchObservedRunningTime="2026-09-11 14:11:43.231315139 +0000 UTC m=+1120.101046402" Sep 11 14:11:43.381357 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:43.381319 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:43.382767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:43.382740 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:11:43.382914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:43.382801 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:11:44.384704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:44.384669 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:44.756411 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:44.756329 2579 patch_prober.go:28] interesting pod/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:11:44.756411 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:44.756384 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:11:45.219335 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:45.219298 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:45.220147 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:45.220088 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:11:45.260784 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:45.260752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:45.353876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:45.353840 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:46.402338 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.402281 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:11:46.476720 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476692 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-socket\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476720 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476722 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-token\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476748 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-certs\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476843 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-credential-socket\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476900 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-podinfo\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476926 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-envoy\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476947 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-data\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.476987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.476974 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istiod-ca-cert\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.477394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477004 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:46.477394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477014 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qpbg8\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-kube-api-access-qpbg8\") pod \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\" (UID: \"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3\") " Sep 11 14:11:46.477394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477121 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:46.477394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477141 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:46.477394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477350 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:46.477683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477474 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:11:46.477683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477540 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.477683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477561 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.477683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477575 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.477683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.477587 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.479832 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.479794 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:11:46.479966 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.479936 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:46.480032 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.479974 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:11:46.480359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.480335 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-kube-api-access-qpbg8" (OuterVolumeSpecName: "kube-api-access-qpbg8") pod "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" (UID: "f3d15ad8-53e7-463b-a8d4-a96896c1e6c3"). InnerVolumeSpecName "kube-api-access-qpbg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:11:46.509890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.509862 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:46.580234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.579802 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.580234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.579832 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.580234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.579848 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.580234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.579861 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpbg8\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-kube-api-access-qpbg8\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.580234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.579875 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:11:46.611261 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.611228 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:46.813887 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:46.813858 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:47.208565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.208452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:47.233068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.233040 2579 generic.go:356] "Generic (PLEG): container finished" podID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerID="698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24" exitCode=0 Sep 11 14:11:47.233265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.233078 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" event={"ID":"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3","Type":"ContainerDied","Data":"698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24"} Sep 11 14:11:47.233265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.233114 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" Sep 11 14:11:47.233265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.233160 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n" event={"ID":"f3d15ad8-53e7-463b-a8d4-a96896c1e6c3","Type":"ContainerDied","Data":"375ee73a11afe639c681c89552cc4ddc171673144180c1e0c47c2bb4ac358d61"} Sep 11 14:11:47.233265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.233193 2579 scope.go:117] "RemoveContainer" containerID="698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24" Sep 11 14:11:47.244284 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.244268 2579 scope.go:117] "RemoveContainer" containerID="a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8" Sep 11 14:11:47.253939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.253923 2579 scope.go:117] "RemoveContainer" containerID="698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24" Sep 11 14:11:47.254212 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:11:47.254194 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24\": container with ID starting with 698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24 not found: ID does not exist" containerID="698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24" Sep 11 14:11:47.254269 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.254219 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24"} err="failed to get container status \"698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24\": rpc error: code = NotFound desc = could not find container \"698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24\": container with ID starting with 698eb30c1ca918040969088bcbc7930c82ee47721f4e3f0bf8e0e91157977d24 not found: ID does not exist" Sep 11 14:11:47.254269 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.254235 2579 scope.go:117] "RemoveContainer" containerID="a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8" Sep 11 14:11:47.254470 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:11:47.254452 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8\": container with ID starting with a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8 not found: ID does not exist" containerID="a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8" Sep 11 14:11:47.254515 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.254476 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8"} err="failed to get container status \"a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8\": rpc error: code = NotFound desc = could not find container \"a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8\": container with ID starting with a3cb1715c7e5a73d7ff37df53f8766182a27e9570c5c94b5b2f7a3b099393fc8 not found: ID does not exist" Sep 11 14:11:47.290124 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.289286 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n"] Sep 11 14:11:47.308634 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.308612 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-80fb45-openshift-default-5bd5d6894b-slk6n"] Sep 11 14:11:47.409364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.409267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:47.643317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:47.643285 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" path="/var/lib/kubelet/pods/f3d15ad8-53e7-463b-a8d4-a96896c1e6c3/volumes" Sep 11 14:11:48.736032 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:48.735985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:49.410162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:49.410125 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:49.485159 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:49.485131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:49.567904 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:49.567869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:49.649036 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:49.649003 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:49.725223 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:49.725155 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:50.068434 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:50.068399 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:50.204492 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:50.204443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:50.936600 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:50.936544 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:56.065772 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.065733 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:56.173746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.173705 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:56.314165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.314135 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8"] Sep 11 14:11:56.314383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.314341 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" containerID="cri-o://43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208" gracePeriod=30 Sep 11 14:11:56.353610 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.353524 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:56.709635 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.709553 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:56.800486 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.800451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:11:56.901851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:11:56.901795 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:01.260227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.260190 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:12:01.261907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.260267 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:12:01.346356 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.346329 2579 generic.go:356] "Generic (PLEG): container finished" podID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerID="55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda" exitCode=0 Sep 11 14:12:01.346455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.346366 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" event={"ID":"637c29a6-56e5-4dc2-b841-78d65be5961f","Type":"ContainerDied","Data":"55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda"} Sep 11 14:12:01.346730 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.346717 2579 scope.go:117] "RemoveContainer" containerID="55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda" Sep 11 14:12:01.347727 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.347697 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:01.347820 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.347784 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:01.347882 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.347831 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:01.578087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.578060 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:12:01.732572 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732543 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faeac48-25e6-4573-bffb-c35cd005ee1e-istiod-ca-cert\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.732759 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732590 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-token\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.732759 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732623 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-credential-socket\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.732759 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732686 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vh77\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-kube-api-access-2vh77\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.732759 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732734 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-data\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.733021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732777 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-certs\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.733021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732799 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-socket\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.733021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732867 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-envoy\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.733021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732897 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-podinfo\") pod \"0faeac48-25e6-4573-bffb-c35cd005ee1e\" (UID: \"0faeac48-25e6-4573-bffb-c35cd005ee1e\") " Sep 11 14:12:01.733021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.732971 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:12:01.733376 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.733073 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:12:01.733376 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.733312 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.733376 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.733342 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.733537 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.733379 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:12:01.733537 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.733447 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:12:01.733537 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.733489 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0faeac48-25e6-4573-bffb-c35cd005ee1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:12:01.735259 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.735233 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:12:01.735357 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.735275 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:12:01.735496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.735479 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:01.735537 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.735495 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-kube-api-access-2vh77" (OuterVolumeSpecName: "kube-api-access-2vh77") pod "0faeac48-25e6-4573-bffb-c35cd005ee1e" (UID: "0faeac48-25e6-4573-bffb-c35cd005ee1e"). InnerVolumeSpecName "kube-api-access-2vh77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834132 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834163 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834173 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834182 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834192 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0faeac48-25e6-4573-bffb-c35cd005ee1e-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834199 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:01.834215 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:01.834207 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vh77\" (UniqueName: \"kubernetes.io/projected/0faeac48-25e6-4573-bffb-c35cd005ee1e-kube-api-access-2vh77\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:12:02.354234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.354207 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" event={"ID":"637c29a6-56e5-4dc2-b841-78d65be5961f","Type":"ContainerStarted","Data":"1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69"} Sep 11 14:12:02.356069 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.356046 2579 generic.go:356] "Generic (PLEG): container finished" podID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerID="43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208" exitCode=0 Sep 11 14:12:02.356195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.356085 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" Sep 11 14:12:02.356195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.356097 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" event={"ID":"0faeac48-25e6-4573-bffb-c35cd005ee1e","Type":"ContainerDied","Data":"43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208"} Sep 11 14:12:02.356195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.356161 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8" event={"ID":"0faeac48-25e6-4573-bffb-c35cd005ee1e","Type":"ContainerDied","Data":"b4187768cf5d560596439a8680aa3270ee84dddc8069eef70dde80e5ea414846"} Sep 11 14:12:02.356195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.356183 2579 scope.go:117] "RemoveContainer" containerID="43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208" Sep 11 14:12:02.366076 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.366058 2579 scope.go:117] "RemoveContainer" containerID="10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c" Sep 11 14:12:02.395174 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.381082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:12:02.395174 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.381144 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:12:02.395174 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.381192 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:12:02.398146 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.398043 2579 scope.go:117] "RemoveContainer" containerID="43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208" Sep 11 14:12:02.398568 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:12:02.398536 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208\": container with ID starting with 43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208 not found: ID does not exist" containerID="43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208" Sep 11 14:12:02.398683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.398579 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208"} err="failed to get container status \"43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208\": rpc error: code = NotFound desc = could not find container \"43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208\": container with ID starting with 43d4665f7574bf6c033b34c1d4216ed584007c889621daaaa8e1ea5bcc060208 not found: ID does not exist" Sep 11 14:12:02.398683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.398602 2579 scope.go:117] "RemoveContainer" containerID="10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c" Sep 11 14:12:02.398993 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:12:02.398970 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c\": container with ID starting with 10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c not found: ID does not exist" containerID="10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c" Sep 11 14:12:02.399083 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.398997 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c"} err="failed to get container status \"10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c\": rpc error: code = NotFound desc = could not find container \"10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c\": container with ID starting with 10470835869dec1e5cf494ae0979c7e607257171b7a26a61c0c1932922f0d52c not found: ID does not exist" Sep 11 14:12:02.401631 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.401613 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8"] Sep 11 14:12:02.409017 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:02.408996 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3e88320d-openshift-default-57674db98d-8bxf8"] Sep 11 14:12:03.004257 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.004225 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.129741 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.129704 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.309703 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.309675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.383806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.383766 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:12:03.384375 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.383829 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:12:03.404128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.404065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:03.643964 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:03.643888 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" path="/var/lib/kubelet/pods/0faeac48-25e6-4573-bffb-c35cd005ee1e/volumes" Sep 11 14:12:04.381768 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:04.381726 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:12:04.381955 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:04.381804 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:12:05.381507 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:05.381471 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:12:05.382001 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:05.381542 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:12:06.381531 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:06.381007 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:12:06.381531 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:06.381067 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:12:07.381903 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:07.381858 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Sep 11 14:12:07.382303 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:07.381924 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Sep 11 14:12:07.965496 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:07.965457 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:08.384645 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:08.384615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:12:08.385012 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:08.384904 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:12:08.385525 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:08.385509 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:12:09.470664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:09.470622 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.089911 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.089876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.444910 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.444830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.560050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.560019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.842304 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842271 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k"] Sep 11 14:12:10.842707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842693 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" Sep 11 14:12:10.842760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842709 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" Sep 11 14:12:10.842760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842718 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" Sep 11 14:12:10.842760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842723 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" Sep 11 14:12:10.842881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842800 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" Sep 11 14:12:10.842881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" Sep 11 14:12:10.842881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842812 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" Sep 11 14:12:10.842881 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.842822 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" Sep 11 14:12:10.846142 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.846117 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:10.849994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.849973 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-261f9c-openshift-default-dockercfg-9r4gt\"" Sep 11 14:12:10.856556 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.856525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:10.863980 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:10.863959 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k"] Sep 11 14:12:11.028863 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.028828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6012c421-2586-43f1-b418-0dcb790731bb-istio-podinfo\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.028871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6012c421-2586-43f1-b418-0dcb790731bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.028891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9hfw\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-kube-api-access-x9hfw\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029034 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.028982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-credential-socket\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029229 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.029052 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-istio-token\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029229 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.029078 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-socket\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029229 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.029125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-certs\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029229 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.029194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-envoy\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.029404 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.029257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-data\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.061788 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.061765 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:11.130006 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.129942 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-envoy\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130006 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.129986 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-data\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130205 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6012c421-2586-43f1-b418-0dcb790731bb-istio-podinfo\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130205 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6012c421-2586-43f1-b418-0dcb790731bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130205 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130087 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9hfw\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-kube-api-access-x9hfw\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130205 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-credential-socket\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130410 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-istio-token\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130410 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130264 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-socket\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130410 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-certs\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130410 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-data\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130635 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130612 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-certs\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130783 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-socket\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130802 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-credential-socket\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.130907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.130786 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6012c421-2586-43f1-b418-0dcb790731bb-istiod-ca-cert\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.132477 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.132456 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-envoy\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.132804 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.132787 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6012c421-2586-43f1-b418-0dcb790731bb-istio-podinfo\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.140989 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.140966 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9hfw\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-kube-api-access-x9hfw\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.141218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.141194 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-istio-token\") pod \"e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.154903 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.154875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:11.156833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.156816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:11.450060 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:12:11.450031 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6012c421_2586_43f1_b418_0dcb790731bb.slice/crio-85dd2a3f8f785fb18714be52edfcc5c91d5cb7d7f7865e388e5efb79e3f38164 WatchSource:0}: Error finding container 85dd2a3f8f785fb18714be52edfcc5c91d5cb7d7f7865e388e5efb79e3f38164: Status 404 returned error can't find the container with id 85dd2a3f8f785fb18714be52edfcc5c91d5cb7d7f7865e388e5efb79e3f38164 Sep 11 14:12:11.451175 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.451154 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k"] Sep 11 14:12:11.452633 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.452592 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:11.452737 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.452678 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:11.452737 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.452723 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:11.483855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.483824 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:11.946300 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:11.946267 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:12.256654 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:12.256546 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:12.448068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:12.448030 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" event={"ID":"6012c421-2586-43f1-b418-0dcb790731bb","Type":"ContainerStarted","Data":"5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e"} Sep 11 14:12:12.448294 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:12.448092 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" event={"ID":"6012c421-2586-43f1-b418-0dcb790731bb","Type":"ContainerStarted","Data":"85dd2a3f8f785fb18714be52edfcc5c91d5cb7d7f7865e388e5efb79e3f38164"} Sep 11 14:12:12.475128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:12.475054 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podStartSLOduration=2.475038848 podStartE2EDuration="2.475038848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:12.471311826 +0000 UTC m=+1149.341043113" watchObservedRunningTime="2026-09-11 14:12:12.475038848 +0000 UTC m=+1149.344770182" Sep 11 14:12:12.575349 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:12.575320 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:13.157261 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:13.157234 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:13.162635 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:13.162613 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:13.454679 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:13.454586 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:13.455755 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:13.455736 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:13.797962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:13.797930 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:14.451920 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:14.451881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:14.558619 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:14.558591 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:14.669407 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:14.669357 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:15.903478 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:15.903417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:16.011897 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:16.011861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:16.174675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:16.174589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:16.609898 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:16.609848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:16.705483 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:16.705441 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:17.170474 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:17.170436 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:18.294981 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:18.294948 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:19.339227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:19.339189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:20.147436 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.147400 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:20.249877 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.249839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:20.381153 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.381094 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:20.726873 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.726836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:20.839050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.839020 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:20.917259 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.917217 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp"] Sep 11 14:12:20.917755 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.917734 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" Sep 11 14:12:20.917840 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.917757 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d15ad8-53e7-463b-a8d4-a96896c1e6c3" containerName="istio-proxy" Sep 11 14:12:20.917840 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.917767 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" Sep 11 14:12:20.917840 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.917774 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0faeac48-25e6-4573-bffb-c35cd005ee1e" containerName="istio-proxy" Sep 11 14:12:20.920244 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.920222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:20.923414 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.923395 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-c52865-dockercfg-lv6xr\"" Sep 11 14:12:20.923567 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.923543 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-c52865\"" Sep 11 14:12:20.935974 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:20.935944 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp"] Sep 11 14:12:21.024836 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.024752 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853e1c77-b259-42b4-a48a-210d5037882d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.024987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.024913 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/853e1c77-b259-42b4-a48a-210d5037882d-kube-api-access-r89vf\") pod \"payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.125702 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.125662 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/853e1c77-b259-42b4-a48a-210d5037882d-kube-api-access-r89vf\") pod \"payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.125856 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.125716 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853e1c77-b259-42b4-a48a-210d5037882d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.126407 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.126389 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853e1c77-b259-42b4-a48a-210d5037882d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.137511 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.137482 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/853e1c77-b259-42b4-a48a-210d5037882d-kube-api-access-r89vf\") pod \"payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.231791 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.231758 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:21.507618 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:21.507589 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp"] Sep 11 14:12:21.508633 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:12:21.508604 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod853e1c77_b259_42b4_a48a_210d5037882d.slice/crio-0f3949df6ab28430704feed4e35efe136ff81700f3d47ff66d69345c24e7d31d WatchSource:0}: Error finding container 0f3949df6ab28430704feed4e35efe136ff81700f3d47ff66d69345c24e7d31d: Status 404 returned error can't find the container with id 0f3949df6ab28430704feed4e35efe136ff81700f3d47ff66d69345c24e7d31d Sep 11 14:12:22.518439 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:22.518403 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" event={"ID":"853e1c77-b259-42b4-a48a-210d5037882d","Type":"ContainerStarted","Data":"6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4"} Sep 11 14:12:22.518826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:22.518457 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" event={"ID":"853e1c77-b259-42b4-a48a-210d5037882d","Type":"ContainerStarted","Data":"0f3949df6ab28430704feed4e35efe136ff81700f3d47ff66d69345c24e7d31d"} Sep 11 14:12:22.518826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:22.518530 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:22.540434 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:22.540386 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" podStartSLOduration=2.54037055 podStartE2EDuration="2.54037055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:22.537357478 +0000 UTC m=+1159.407088739" watchObservedRunningTime="2026-09-11 14:12:22.54037055 +0000 UTC m=+1159.410101812" Sep 11 14:12:24.257523 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:24.257480 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:29.497296 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:29.497252 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Sep 11 14:12:29.497728 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:29.497331 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Sep 11 14:12:31.575539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:31.575507 2579 generic.go:356] "Generic (PLEG): container finished" podID="6012c421-2586-43f1-b418-0dcb790731bb" containerID="5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e" exitCode=0 Sep 11 14:12:31.575975 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:31.575589 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" event={"ID":"6012c421-2586-43f1-b418-0dcb790731bb","Type":"ContainerDied","Data":"5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e"} Sep 11 14:12:31.576020 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:31.576010 2579 scope.go:117] "RemoveContainer" containerID="5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e" Sep 11 14:12:31.577166 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:31.577092 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:31.577280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:31.577231 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:31.577343 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:31.577281 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:32.588263 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:32.588225 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" event={"ID":"6012c421-2586-43f1-b418-0dcb790731bb","Type":"ContainerStarted","Data":"bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36"} Sep 11 14:12:33.157341 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:33.157309 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:33.157549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:33.157479 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:12:33.157549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:33.157530 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:12:33.526760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:33.526674 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:12:34.157951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:34.157913 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:12:34.158466 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:34.157978 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:12:35.157259 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:35.157226 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:12:35.157430 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:35.157280 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:12:36.157985 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:36.157949 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:12:36.158403 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:36.158005 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:12:37.132905 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:37.132861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:37.157847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:37.157822 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Sep 11 14:12:37.157993 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:37.157873 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Sep 11 14:12:38.161644 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:38.161614 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:38.162091 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:38.162072 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:38.162842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:38.162825 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:12:38.563437 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:38.563406 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:39.316127 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:39.316067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:39.417275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:39.417241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:39.847938 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:39.847891 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:40.079792 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:40.079757 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:41.106748 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:41.106708 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:42.037852 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:42.037820 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:43.065591 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:43.065557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:43.830622 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:43.830590 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:44.721115 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:44.721052 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:45.467200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:45.467167 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:45.890907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:45.890856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.224855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.224758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.516981 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.516910 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx"] Sep 11 14:12:46.519825 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.519804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.524388 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.524368 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5cf565-openshift-default-dockercfg-gcrp8\"" Sep 11 14:12:46.534445 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.534423 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx"] Sep 11 14:12:46.546484 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.546460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:46.686369 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-podinfo\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686548 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686376 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-data\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686548 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686404 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6skw4\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-kube-api-access-6skw4\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686548 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686472 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-envoy\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686544 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-credential-socket\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686599 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-token\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-socket\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686692 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-certs\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.686878 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.686716 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787307 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-socket\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787307 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787257 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-certs\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787307 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787274 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-podinfo\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787352 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-data\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6skw4\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-kube-api-access-6skw4\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787512 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-envoy\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787799 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-credential-socket\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787799 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-token\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787799 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787686 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-socket\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.787799 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787735 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-certs\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.788003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787854 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-data\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.788003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.787987 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-credential-socket\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.788136 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.788066 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istiod-ca-cert\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.789931 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.789910 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-envoy\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.790036 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.789968 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-podinfo\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.795480 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.795462 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6skw4\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-kube-api-access-6skw4\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.796004 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.795989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-token\") pod \"e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:46.830880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:46.830857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:47.124415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.124388 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx"] Sep 11 14:12:47.125639 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:12:47.125612 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd865c57c_3c7a_4d7c_b1fa_78163296f3ad.slice/crio-4fd44b7ed44b5032b87fd70d02421502d31272c5c436c710ee15a6f0406372e7 WatchSource:0}: Error finding container 4fd44b7ed44b5032b87fd70d02421502d31272c5c436c710ee15a6f0406372e7: Status 404 returned error can't find the container with id 4fd44b7ed44b5032b87fd70d02421502d31272c5c436c710ee15a6f0406372e7 Sep 11 14:12:47.128807 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.128762 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:47.128923 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.128876 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:47.128923 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.128918 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:12:47.207214 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.207182 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:47.684171 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.684135 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" event={"ID":"d865c57c-3c7a-4d7c-b1fa-78163296f3ad","Type":"ContainerStarted","Data":"a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985"} Sep 11 14:12:47.684337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.684189 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" event={"ID":"d865c57c-3c7a-4d7c-b1fa-78163296f3ad","Type":"ContainerStarted","Data":"4fd44b7ed44b5032b87fd70d02421502d31272c5c436c710ee15a6f0406372e7"} Sep 11 14:12:47.710844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.710786 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podStartSLOduration=1.7107712830000001 podStartE2EDuration="1.710771283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:47.709965188 +0000 UTC m=+1184.579696450" watchObservedRunningTime="2026-09-11 14:12:47.710771283 +0000 UTC m=+1184.580502545" Sep 11 14:12:47.831307 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:47.831270 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:48.836661 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:48.836636 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:49.019590 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:49.019550 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:49.672353 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:49.672308 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:49.696245 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:49.696213 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:49.697087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:49.697066 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:12:49.765233 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:49.765190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:50.126560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:50.126522 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:51.071402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:51.071361 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:51.913378 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:51.913330 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:53.289066 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:53.289026 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:53.387427 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:53.387386 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:53.827315 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:53.827281 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:53.930851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:53.930814 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:54.004501 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.004469 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n"] Sep 11 14:12:54.007734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.007713 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.010405 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.010385 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5cf565\"" Sep 11 14:12:54.010526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.010473 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5cf565-dockercfg-2pp4g\"" Sep 11 14:12:54.025789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.025761 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n"] Sep 11 14:12:54.058022 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.057984 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjwtg\" (UniqueName: \"kubernetes.io/projected/c2d775cc-2682-418e-a873-9ea29769f6f8-kube-api-access-mjwtg\") pod \"payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.058248 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.058093 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d775cc-2682-418e-a873-9ea29769f6f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.159165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.159056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjwtg\" (UniqueName: \"kubernetes.io/projected/c2d775cc-2682-418e-a873-9ea29769f6f8-kube-api-access-mjwtg\") pod \"payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.159165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.159147 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d775cc-2682-418e-a873-9ea29769f6f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.159770 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.159751 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d775cc-2682-418e-a873-9ea29769f6f8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.173085 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.173064 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjwtg\" (UniqueName: \"kubernetes.io/projected/c2d775cc-2682-418e-a873-9ea29769f6f8-kube-api-access-mjwtg\") pod \"payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.318623 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.318589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.597972 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:12:54.597947 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d775cc_2682_418e_a873_9ea29769f6f8.slice/crio-5252c594ce640d4ac4aa6a63a4db6bb67c4cc383998194cb1875f3590f63d144 WatchSource:0}: Error finding container 5252c594ce640d4ac4aa6a63a4db6bb67c4cc383998194cb1875f3590f63d144: Status 404 returned error can't find the container with id 5252c594ce640d4ac4aa6a63a4db6bb67c4cc383998194cb1875f3590f63d144 Sep 11 14:12:54.599194 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.599170 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n"] Sep 11 14:12:54.729469 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.729430 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" event={"ID":"c2d775cc-2682-418e-a873-9ea29769f6f8","Type":"ContainerStarted","Data":"65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922"} Sep 11 14:12:54.729649 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.729492 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" event={"ID":"c2d775cc-2682-418e-a873-9ea29769f6f8","Type":"ContainerStarted","Data":"5252c594ce640d4ac4aa6a63a4db6bb67c4cc383998194cb1875f3590f63d144"} Sep 11 14:12:54.729649 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.729520 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:12:54.754073 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:54.754011 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" podStartSLOduration=1.7539928009999999 podStartE2EDuration="1.753992801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:12:54.752790912 +0000 UTC m=+1191.622522175" watchObservedRunningTime="2026-09-11 14:12:54.753992801 +0000 UTC m=+1191.623724064" Sep 11 14:12:58.256568 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:58.256521 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:12:58.566347 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:12:58.566313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:05.737715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:05.737684 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:13:08.843557 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:08.843518 2579 generic.go:356] "Generic (PLEG): container finished" podID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerID="a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985" exitCode=0 Sep 11 14:13:08.844013 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:08.843592 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" event={"ID":"d865c57c-3c7a-4d7c-b1fa-78163296f3ad","Type":"ContainerDied","Data":"a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985"} Sep 11 14:13:08.844013 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:08.843966 2579 scope.go:117] "RemoveContainer" containerID="a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985" Sep 11 14:13:08.845168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:08.845137 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:08.845288 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:08.845221 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:08.845288 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:08.845266 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:09.851656 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:09.851625 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" event={"ID":"d865c57c-3c7a-4d7c-b1fa-78163296f3ad","Type":"ContainerStarted","Data":"3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a"} Sep 11 14:13:10.831614 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:10.831578 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:13:10.831808 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:10.831787 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:13:10.831868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:10.831847 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:13:11.832250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:11.832213 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:13:11.832664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:11.832295 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:13:12.831553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:12.831516 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:13:12.831739 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:12.831585 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:13:13.831612 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:13.831578 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:13:13.832133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:13.831643 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:13:14.307524 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:14.307490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:14.831685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:14.831650 2579 patch_prober.go:28] interesting pod/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Sep 11 14:13:14.832069 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:14.831741 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Sep 11 14:13:15.835449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:15.835422 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:13:15.835822 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:15.835731 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:13:15.836355 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:15.836339 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:13:16.112598 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:16.112503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:17.104081 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:17.104045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:17.276982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:17.276938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:18.356154 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:18.356091 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:19.499471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:19.499433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:19.715370 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:19.715337 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:20.485804 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:20.485767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:30.705532 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:30.705490 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:30.994651 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:30.994575 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl"] Sep 11 14:13:30.997892 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:30.997872 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.005109 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.005083 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d630fb00-openshift-default-dockercfg-jpxnz\"" Sep 11 14:13:31.014291 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.014263 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl"] Sep 11 14:13:31.039482 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039455 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba635564-2081-4c62-9fb0-f96d16167263-istiod-ca-cert\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.039731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039502 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-istio-token\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.039731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039565 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-socket\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.039731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039717 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvwkd\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-kube-api-access-vvwkd\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.039957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039795 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-certs\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.039957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-data\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.040162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.039950 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-envoy\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.040162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.040007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-credential-socket\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.040162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.040054 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba635564-2081-4c62-9fb0-f96d16167263-istio-podinfo\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.040966 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.040935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:31.141386 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-data\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141386 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141390 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-envoy\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-credential-socket\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141548 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba635564-2081-4c62-9fb0-f96d16167263-istio-podinfo\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141615 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba635564-2081-4c62-9fb0-f96d16167263-istiod-ca-cert\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141654 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-istio-token\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141700 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-socket\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141762 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvwkd\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-kube-api-access-vvwkd\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141774 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-credential-socket\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.141806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-certs\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.142075 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.141807 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-data\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.142162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.142129 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-certs\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.142162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.142127 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-socket\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.142578 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.142560 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba635564-2081-4c62-9fb0-f96d16167263-istiod-ca-cert\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.143823 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.143805 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-envoy\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.144588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.144565 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba635564-2081-4c62-9fb0-f96d16167263-istio-podinfo\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.152845 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.152822 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-istio-token\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.153332 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.153302 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvwkd\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-kube-api-access-vvwkd\") pod \"e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.309196 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.309163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:31.594007 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.593977 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl"] Sep 11 14:13:31.594614 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:13:31.594590 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba635564_2081_4c62_9fb0_f96d16167263.slice/crio-6118e6fed265efa8d6a0ee40ff3f3b29b9f838422f5e9c85410bb2aa19c02a9e WatchSource:0}: Error finding container 6118e6fed265efa8d6a0ee40ff3f3b29b9f838422f5e9c85410bb2aa19c02a9e: Status 404 returned error can't find the container with id 6118e6fed265efa8d6a0ee40ff3f3b29b9f838422f5e9c85410bb2aa19c02a9e Sep 11 14:13:31.597082 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.597048 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:31.597200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.597167 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:31.597259 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.597213 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:31.779896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:31.779861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:32.007495 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:32.007394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" event={"ID":"ba635564-2081-4c62-9fb0-f96d16167263","Type":"ContainerStarted","Data":"77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9"} Sep 11 14:13:32.007495 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:32.007493 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" event={"ID":"ba635564-2081-4c62-9fb0-f96d16167263","Type":"ContainerStarted","Data":"6118e6fed265efa8d6a0ee40ff3f3b29b9f838422f5e9c85410bb2aa19c02a9e"} Sep 11 14:13:32.038386 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:32.038332 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podStartSLOduration=2.038316773 podStartE2EDuration="2.038316773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:32.037336909 +0000 UTC m=+1228.907068170" watchObservedRunningTime="2026-09-11 14:13:32.038316773 +0000 UTC m=+1228.908048035" Sep 11 14:13:32.309654 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:32.309615 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:32.315025 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:32.315000 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:33.013590 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:33.013559 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:33.014523 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:33.014505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:33.429255 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:33.429210 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:33.984455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:33.984420 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:34.214367 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:34.214331 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:34.763810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:34.763763 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:35.642413 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:35.642376 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:36.867040 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:36.866966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:37.865913 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:37.865856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:38.826743 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:38.826700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:38.924688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:38.924657 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:39.038002 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.037966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:39.536206 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.536166 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:39.650838 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.650794 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:13:39.761196 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.761164 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx"] Sep 11 14:13:39.764011 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.763988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:39.782191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.781742 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d630fb00-dockercfg-v2pmw\"" Sep 11 14:13:39.782191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.781963 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d630fb00\"" Sep 11 14:13:39.789734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.788722 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx"] Sep 11 14:13:39.837732 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.837694 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a277570-261f-4e02-a2a5-7ef1f3244ee3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:39.838359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.838334 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84wtm\" (UniqueName: \"kubernetes.io/projected/4a277570-261f-4e02-a2a5-7ef1f3244ee3-kube-api-access-84wtm\") pod \"payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:39.939872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.939840 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a277570-261f-4e02-a2a5-7ef1f3244ee3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:39.940030 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.939885 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84wtm\" (UniqueName: \"kubernetes.io/projected/4a277570-261f-4e02-a2a5-7ef1f3244ee3-kube-api-access-84wtm\") pod \"payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:39.940646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.940627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a277570-261f-4e02-a2a5-7ef1f3244ee3-plugins-config-volume\") pod \"payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:39.950713 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:39.950684 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84wtm\" (UniqueName: \"kubernetes.io/projected/4a277570-261f-4e02-a2a5-7ef1f3244ee3-kube-api-access-84wtm\") pod \"payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:40.099981 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:40.099934 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:40.420023 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:13:40.419991 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a277570_261f_4e02_a2a5_7ef1f3244ee3.slice/crio-ffa819d22c5dacb8b53580c599b15aa18a137ab1783331689644bf74469e0b5b WatchSource:0}: Error finding container ffa819d22c5dacb8b53580c599b15aa18a137ab1783331689644bf74469e0b5b: Status 404 returned error can't find the container with id ffa819d22c5dacb8b53580c599b15aa18a137ab1783331689644bf74469e0b5b Sep 11 14:13:40.423522 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:40.423502 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx"] Sep 11 14:13:41.070623 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:41.070590 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" event={"ID":"4a277570-261f-4e02-a2a5-7ef1f3244ee3","Type":"ContainerStarted","Data":"8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23"} Sep 11 14:13:41.071130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:41.070679 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" event={"ID":"4a277570-261f-4e02-a2a5-7ef1f3244ee3","Type":"ContainerStarted","Data":"ffa819d22c5dacb8b53580c599b15aa18a137ab1783331689644bf74469e0b5b"} Sep 11 14:13:41.071130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:41.070735 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:41.096890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:41.096809 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" podStartSLOduration=2.096785187 podStartE2EDuration="2.096785187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:13:41.094606839 +0000 UTC m=+1237.964338097" watchObservedRunningTime="2026-09-11 14:13:41.096785187 +0000 UTC m=+1237.966516450" Sep 11 14:13:52.092127 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:52.091262 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:13:57.184308 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:57.184279 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba635564-2081-4c62-9fb0-f96d16167263" containerID="77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9" exitCode=0 Sep 11 14:13:57.184802 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:57.184345 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" event={"ID":"ba635564-2081-4c62-9fb0-f96d16167263","Type":"ContainerDied","Data":"77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9"} Sep 11 14:13:57.184802 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:57.184790 2579 scope.go:117] "RemoveContainer" containerID="77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9" Sep 11 14:13:57.185918 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:57.185893 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:57.185967 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:57.185959 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:57.186003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:57.185990 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:13:58.197392 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:58.197353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" event={"ID":"ba635564-2081-4c62-9fb0-f96d16167263","Type":"ContainerStarted","Data":"14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087"} Sep 11 14:13:58.310322 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:58.310277 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:13:58.311619 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:58.311593 2579 patch_prober.go:28] interesting pod/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:13:58.311743 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:58.311642 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 11 14:13:59.309702 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:59.309660 2579 patch_prober.go:28] interesting pod/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:13:59.310236 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:13:59.309739 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 11 14:14:00.310423 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:00.310388 2579 patch_prober.go:28] interesting pod/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:14:00.310836 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:00.310441 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 11 14:14:01.310445 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:01.310407 2579 patch_prober.go:28] interesting pod/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:14:01.310852 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:01.310462 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 11 14:14:02.309961 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:02.309924 2579 patch_prober.go:28] interesting pod/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:14:02.310213 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:02.309997 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 11 14:14:03.013948 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:03.013912 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:14:03.309940 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:03.309896 2579 patch_prober.go:28] interesting pod/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Sep 11 14:14:03.310179 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:03.309980 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Sep 11 14:14:03.876767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:03.876734 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:04.313436 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:04.313406 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:14:04.314449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:04.314430 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:14:05.432145 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:05.432114 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:06.460293 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:06.460255 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:07.557131 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:07.557060 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:08.865694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:08.865650 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:08.998884 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:08.998848 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:09.647497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:09.647461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:09.758337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:09.758294 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:09.938584 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:09.938495 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:38.277940 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.277903 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw"] Sep 11 14:14:38.280448 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.280432 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:38.286188 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.286167 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:14:38.286463 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.286440 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:14:38.294093 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.294067 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw"] Sep 11 14:14:38.384080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.384049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb125f2-5c31-444b-8888-770d6b915df1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw\" (UID: \"5eb125f2-5c31-444b-8888-770d6b915df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:38.484749 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.484715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb125f2-5c31-444b-8888-770d6b915df1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw\" (UID: \"5eb125f2-5c31-444b-8888-770d6b915df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:38.485336 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.485316 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb125f2-5c31-444b-8888-770d6b915df1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw\" (UID: \"5eb125f2-5c31-444b-8888-770d6b915df1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:38.592476 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.592444 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:38.869016 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:38.868989 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw"] Sep 11 14:14:38.869160 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:14:38.869029 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb125f2_5c31_444b_8888_770d6b915df1.slice/crio-e97af405529dc45aaa8ff0a8e541c4a10d6efd248f91af04dd86cfd25be2cf80 WatchSource:0}: Error finding container e97af405529dc45aaa8ff0a8e541c4a10d6efd248f91af04dd86cfd25be2cf80: Status 404 returned error can't find the container with id e97af405529dc45aaa8ff0a8e541c4a10d6efd248f91af04dd86cfd25be2cf80 Sep 11 14:14:39.283424 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:39.283384 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:39.487808 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:39.487773 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" event={"ID":"5eb125f2-5c31-444b-8888-770d6b915df1","Type":"ContainerStarted","Data":"c77dc75059452e461180974b15ba77bfc8c518cf6c1624779a4f570fb10ec009"} Sep 11 14:14:39.487958 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:39.487839 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" event={"ID":"5eb125f2-5c31-444b-8888-770d6b915df1","Type":"ContainerStarted","Data":"e97af405529dc45aaa8ff0a8e541c4a10d6efd248f91af04dd86cfd25be2cf80"} Sep 11 14:14:39.521369 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:39.521318 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" podStartSLOduration=1.521302778 podStartE2EDuration="1.521302778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:14:39.518486996 +0000 UTC m=+1296.388218258" watchObservedRunningTime="2026-09-11 14:14:39.521302778 +0000 UTC m=+1296.391034094" Sep 11 14:14:40.150856 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:40.150817 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:40.496040 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:40.495944 2579 generic.go:356] "Generic (PLEG): container finished" podID="5eb125f2-5c31-444b-8888-770d6b915df1" containerID="c77dc75059452e461180974b15ba77bfc8c518cf6c1624779a4f570fb10ec009" exitCode=0 Sep 11 14:14:40.496416 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:40.496032 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" event={"ID":"5eb125f2-5c31-444b-8888-770d6b915df1","Type":"ContainerDied","Data":"c77dc75059452e461180974b15ba77bfc8c518cf6c1624779a4f570fb10ec009"} Sep 11 14:14:40.998662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:40.998623 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:41.636185 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:41.636160 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:41.716535 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:41.716501 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb125f2-5c31-444b-8888-770d6b915df1-maas-api-service-ca\") pod \"5eb125f2-5c31-444b-8888-770d6b915df1\" (UID: \"5eb125f2-5c31-444b-8888-770d6b915df1\") " Sep 11 14:14:41.716917 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:41.716896 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb125f2-5c31-444b-8888-770d6b915df1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5eb125f2-5c31-444b-8888-770d6b915df1" (UID: "5eb125f2-5c31-444b-8888-770d6b915df1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:14:41.817996 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:41.817962 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb125f2-5c31-444b-8888-770d6b915df1-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:14:41.893645 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:41.893615 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:42.509762 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:42.509727 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" event={"ID":"5eb125f2-5c31-444b-8888-770d6b915df1","Type":"ContainerDied","Data":"e97af405529dc45aaa8ff0a8e541c4a10d6efd248f91af04dd86cfd25be2cf80"} Sep 11 14:14:42.509962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:42.509771 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw" Sep 11 14:14:42.509962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:42.509774 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e97af405529dc45aaa8ff0a8e541c4a10d6efd248f91af04dd86cfd25be2cf80" Sep 11 14:14:42.852761 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:42.852721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:43.822623 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:43.822588 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:53.991003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:53.990949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:54.036776 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.036742 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx"] Sep 11 14:14:54.036993 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.036968 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" podUID="4a277570-261f-4e02-a2a5-7ef1f3244ee3" containerName="payload-processing" containerID="cri-o://8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23" gracePeriod=30 Sep 11 14:14:54.117200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.117168 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:54.216987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.216849 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:54.294008 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.293903 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw"] Sep 11 14:14:54.298290 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.298263 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d630fb00-qvtnw"] Sep 11 14:14:54.304220 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.304194 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:54.326851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.326828 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:14:54.460094 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.460062 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84wtm\" (UniqueName: \"kubernetes.io/projected/4a277570-261f-4e02-a2a5-7ef1f3244ee3-kube-api-access-84wtm\") pod \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " Sep 11 14:14:54.460094 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.460109 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a277570-261f-4e02-a2a5-7ef1f3244ee3-plugins-config-volume\") pod \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\" (UID: \"4a277570-261f-4e02-a2a5-7ef1f3244ee3\") " Sep 11 14:14:54.460546 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.460521 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a277570-261f-4e02-a2a5-7ef1f3244ee3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4a277570-261f-4e02-a2a5-7ef1f3244ee3" (UID: "4a277570-261f-4e02-a2a5-7ef1f3244ee3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:14:54.462357 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.462337 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a277570-261f-4e02-a2a5-7ef1f3244ee3-kube-api-access-84wtm" (OuterVolumeSpecName: "kube-api-access-84wtm") pod "4a277570-261f-4e02-a2a5-7ef1f3244ee3" (UID: "4a277570-261f-4e02-a2a5-7ef1f3244ee3"). InnerVolumeSpecName "kube-api-access-84wtm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:14:54.561339 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.561295 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84wtm\" (UniqueName: \"kubernetes.io/projected/4a277570-261f-4e02-a2a5-7ef1f3244ee3-kube-api-access-84wtm\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:14:54.561339 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.561325 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a277570-261f-4e02-a2a5-7ef1f3244ee3-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:14:54.586422 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.586384 2579 generic.go:356] "Generic (PLEG): container finished" podID="4a277570-261f-4e02-a2a5-7ef1f3244ee3" containerID="8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23" exitCode=0 Sep 11 14:14:54.586569 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.586427 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" Sep 11 14:14:54.586569 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.586461 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" event={"ID":"4a277570-261f-4e02-a2a5-7ef1f3244ee3","Type":"ContainerDied","Data":"8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23"} Sep 11 14:14:54.586569 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.586516 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx" event={"ID":"4a277570-261f-4e02-a2a5-7ef1f3244ee3","Type":"ContainerDied","Data":"ffa819d22c5dacb8b53580c599b15aa18a137ab1783331689644bf74469e0b5b"} Sep 11 14:14:54.586569 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.586539 2579 scope.go:117] "RemoveContainer" containerID="8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23" Sep 11 14:14:54.597056 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.597038 2579 scope.go:117] "RemoveContainer" containerID="8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23" Sep 11 14:14:54.597381 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:14:54.597360 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23\": container with ID starting with 8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23 not found: ID does not exist" containerID="8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23" Sep 11 14:14:54.597456 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.597388 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23"} err="failed to get container status \"8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23\": rpc error: code = NotFound desc = could not find container \"8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23\": container with ID starting with 8d2a006de6af23a21245ed42fe497d4ad5bbcbf880f991155dbccd039ca44a23 not found: ID does not exist" Sep 11 14:14:54.620482 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.620461 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx"] Sep 11 14:14:54.625188 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.625165 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d630fb00-8bdb68ccc-652dx"] Sep 11 14:14:54.743997 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:54.743963 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:55.643821 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:55.643785 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a277570-261f-4e02-a2a5-7ef1f3244ee3" path="/var/lib/kubelet/pods/4a277570-261f-4e02-a2a5-7ef1f3244ee3/volumes" Sep 11 14:14:55.644230 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:55.644144 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb125f2-5c31-444b-8888-770d6b915df1" path="/var/lib/kubelet/pods/5eb125f2-5c31-444b-8888-770d6b915df1/volumes" Sep 11 14:14:55.810674 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:55.810643 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:56.955992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:56.955957 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:57.398234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:57.398202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:57.496523 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:57.496479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:57.835561 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:57.835526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:14:58.653863 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:14:58.653826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:00.239799 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.239760 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr"] Sep 11 14:15:00.240472 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.240449 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a277570-261f-4e02-a2a5-7ef1f3244ee3" containerName="payload-processing" Sep 11 14:15:00.240582 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.240475 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a277570-261f-4e02-a2a5-7ef1f3244ee3" containerName="payload-processing" Sep 11 14:15:00.240582 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.240487 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb125f2-5c31-444b-8888-770d6b915df1" containerName="revoke-keys" Sep 11 14:15:00.240582 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.240495 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb125f2-5c31-444b-8888-770d6b915df1" containerName="revoke-keys" Sep 11 14:15:00.240754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.240638 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a277570-261f-4e02-a2a5-7ef1f3244ee3" containerName="payload-processing" Sep 11 14:15:00.240754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.240653 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb125f2-5c31-444b-8888-770d6b915df1" containerName="revoke-keys" Sep 11 14:15:00.243721 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.243699 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t"] Sep 11 14:15:00.243852 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.243814 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" Sep 11 14:15:00.248560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.248539 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x"] Sep 11 14:15:00.248683 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.248648 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" Sep 11 14:15:00.251064 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.251048 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" Sep 11 14:15:00.263473 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.263451 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:15:00.266890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.266867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" Sep 11 14:15:00.269294 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.268887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" Sep 11 14:15:00.274897 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.274881 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" Sep 11 14:15:00.320156 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.320121 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x"] Sep 11 14:15:00.330350 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.330316 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr"] Sep 11 14:15:00.330668 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.330649 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t"] Sep 11 14:15:00.617120 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:15:00.617075 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec718bd0_4d61_4e65_9440_d31ff3db3d96.slice/crio-e63f6812ae79d2db074bda1e203b1bf25dd2374b67ff15e26c9c89349e09afa7 WatchSource:0}: Error finding container e63f6812ae79d2db074bda1e203b1bf25dd2374b67ff15e26c9c89349e09afa7: Status 404 returned error can't find the container with id e63f6812ae79d2db074bda1e203b1bf25dd2374b67ff15e26c9c89349e09afa7 Sep 11 14:15:00.617120 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.617114 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr"] Sep 11 14:15:00.628091 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.628068 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x"] Sep 11 14:15:00.631715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.631693 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" event={"ID":"ec718bd0-4d61-4e65-9440-d31ff3db3d96","Type":"ContainerStarted","Data":"e63f6812ae79d2db074bda1e203b1bf25dd2374b67ff15e26c9c89349e09afa7"} Sep 11 14:15:00.634359 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:15:00.634320 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd3db8b_37ee_4a7a_9d77_215d0ff669ce.slice/crio-7c15eef3c1d336e41cde31ab497bcac16462c7a3339f77f5b39854e3faf0e10c WatchSource:0}: Error finding container 7c15eef3c1d336e41cde31ab497bcac16462c7a3339f77f5b39854e3faf0e10c: Status 404 returned error can't find the container with id 7c15eef3c1d336e41cde31ab497bcac16462c7a3339f77f5b39854e3faf0e10c Sep 11 14:15:00.689373 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:00.689351 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t"] Sep 11 14:15:00.690333 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:15:00.690314 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d70d6c_5209_43e0_a363_7f18858d7bfb.slice/crio-168a04795ab33c9df053118ee5af7cb4e46bb9b25ef2667121921ffbf92587ba WatchSource:0}: Error finding container 168a04795ab33c9df053118ee5af7cb4e46bb9b25ef2667121921ffbf92587ba: Status 404 returned error can't find the container with id 168a04795ab33c9df053118ee5af7cb4e46bb9b25ef2667121921ffbf92587ba Sep 11 14:15:01.640284 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.640251 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebd3db8b-37ee-4a7a-9d77-215d0ff669ce" containerID="648172f4c6ef4f205d297f40a054bf604e30ea4821d155048545e2795e419bb2" exitCode=0 Sep 11 14:15:01.643793 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.643769 2579 generic.go:356] "Generic (PLEG): container finished" podID="97d70d6c-5209-43e0-a363-7f18858d7bfb" containerID="9ddf40d7e7d8b3fc2a1998002affec7f1d88706bb910ddd6a91fd5249e25bb45" exitCode=0 Sep 11 14:15:01.643995 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.643976 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" event={"ID":"ebd3db8b-37ee-4a7a-9d77-215d0ff669ce","Type":"ContainerDied","Data":"648172f4c6ef4f205d297f40a054bf604e30ea4821d155048545e2795e419bb2"} Sep 11 14:15:01.644080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.644022 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" event={"ID":"ebd3db8b-37ee-4a7a-9d77-215d0ff669ce","Type":"ContainerStarted","Data":"7c15eef3c1d336e41cde31ab497bcac16462c7a3339f77f5b39854e3faf0e10c"} Sep 11 14:15:01.644080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.644048 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" event={"ID":"ec718bd0-4d61-4e65-9440-d31ff3db3d96","Type":"ContainerStarted","Data":"3b8cca7d7394b0679c36a91d0cec2ca2bbb134261f4fb864282734d1c952124d"} Sep 11 14:15:01.644080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.644077 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" event={"ID":"97d70d6c-5209-43e0-a363-7f18858d7bfb","Type":"ContainerDied","Data":"9ddf40d7e7d8b3fc2a1998002affec7f1d88706bb910ddd6a91fd5249e25bb45"} Sep 11 14:15:01.644235 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.644134 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" event={"ID":"97d70d6c-5209-43e0-a363-7f18858d7bfb","Type":"ContainerStarted","Data":"168a04795ab33c9df053118ee5af7cb4e46bb9b25ef2667121921ffbf92587ba"} Sep 11 14:15:01.698088 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:01.698034 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" podStartSLOduration=1.698016812 podStartE2EDuration="1.698016812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:15:01.696136446 +0000 UTC m=+1318.565867745" watchObservedRunningTime="2026-09-11 14:15:01.698016812 +0000 UTC m=+1318.567748115" Sep 11 14:15:02.572570 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:02.572528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:02.652033 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:02.652001 2579 generic.go:356] "Generic (PLEG): container finished" podID="ec718bd0-4d61-4e65-9440-d31ff3db3d96" containerID="3b8cca7d7394b0679c36a91d0cec2ca2bbb134261f4fb864282734d1c952124d" exitCode=0 Sep 11 14:15:02.652492 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:02.652093 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" event={"ID":"ec718bd0-4d61-4e65-9440-d31ff3db3d96","Type":"ContainerDied","Data":"3b8cca7d7394b0679c36a91d0cec2ca2bbb134261f4fb864282734d1c952124d"} Sep 11 14:15:02.794401 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:02.794368 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" Sep 11 14:15:02.842457 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:02.842429 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" Sep 11 14:15:03.659493 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.659473 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" Sep 11 14:15:03.659851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.659505 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t" event={"ID":"97d70d6c-5209-43e0-a363-7f18858d7bfb","Type":"ContainerDied","Data":"168a04795ab33c9df053118ee5af7cb4e46bb9b25ef2667121921ffbf92587ba"} Sep 11 14:15:03.659851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.659547 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="168a04795ab33c9df053118ee5af7cb4e46bb9b25ef2667121921ffbf92587ba" Sep 11 14:15:03.660840 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.660823 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" Sep 11 14:15:03.660938 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.660834 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x" event={"ID":"ebd3db8b-37ee-4a7a-9d77-215d0ff669ce","Type":"ContainerDied","Data":"7c15eef3c1d336e41cde31ab497bcac16462c7a3339f77f5b39854e3faf0e10c"} Sep 11 14:15:03.660938 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.660876 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c15eef3c1d336e41cde31ab497bcac16462c7a3339f77f5b39854e3faf0e10c" Sep 11 14:15:03.785865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.785843 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" Sep 11 14:15:03.842760 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.842726 2579 scope.go:117] "RemoveContainer" containerID="f39b39652bf8faec4a92bd2a477234384d909b44c40b1cc50de276d996f9729c" Sep 11 14:15:03.853316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.853297 2579 scope.go:117] "RemoveContainer" containerID="0e117d599925fc2f6480c06ffdbda3bcde202d6971040b78b4e734685ce5059c" Sep 11 14:15:03.862368 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:03.862348 2579 scope.go:117] "RemoveContainer" containerID="f5ca31488ecbf4bc0289fbf6a40a3f2a93979f87155af8e2ce7652bc75ea5dce" Sep 11 14:15:04.669576 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:04.669529 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" event={"ID":"ec718bd0-4d61-4e65-9440-d31ff3db3d96","Type":"ContainerDied","Data":"e63f6812ae79d2db074bda1e203b1bf25dd2374b67ff15e26c9c89349e09afa7"} Sep 11 14:15:04.669576 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:04.669572 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr" Sep 11 14:15:04.670003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:04.669588 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e63f6812ae79d2db074bda1e203b1bf25dd2374b67ff15e26c9c89349e09afa7" Sep 11 14:15:33.272546 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:33.272452 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:40.405224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:40.405186 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:40.505571 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:40.505533 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:46.119756 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:46.119703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:15:58.795825 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:15:58.795769 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:12.130833 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:12.130796 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:13.363422 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:13.363382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.839181 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:21.839146 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:21.929820 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:21.929775 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:22.147130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:22.147037 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl"] Sep 11 14:16:22.147328 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:22.147303 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" containerID="cri-o://14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087" gracePeriod=30 Sep 11 14:16:22.161588 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:22.161556 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:24.727141 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:24.727081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:24.843653 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:24.843616 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:25.839905 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:25.839875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:27.405331 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.405277 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:16:27.520037 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520010 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-credential-socket\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520179 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520065 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-certs\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520179 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520121 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-data\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520179 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520152 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba635564-2081-4c62-9fb0-f96d16167263-istiod-ca-cert\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520179 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520175 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba635564-2081-4c62-9fb0-f96d16167263-istio-podinfo\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520281 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-istio-token\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520328 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvwkd\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-kube-api-access-vvwkd\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520321 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:27.520417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-socket\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520405 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-envoy\") pod \"ba635564-2081-4c62-9fb0-f96d16167263\" (UID: \"ba635564-2081-4c62-9fb0-f96d16167263\") " Sep 11 14:16:27.520417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520404 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:27.520719 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520420 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:27.520719 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520613 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba635564-2081-4c62-9fb0-f96d16167263-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:16:27.520719 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520704 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:27.520879 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520820 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.520879 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520839 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.520879 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520853 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.520879 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520867 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.521021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.520879 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba635564-2081-4c62-9fb0-f96d16167263-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.522549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.522523 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:27.522655 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.522574 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-kube-api-access-vvwkd" (OuterVolumeSpecName: "kube-api-access-vvwkd") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "kube-api-access-vvwkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:16:27.522996 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.522977 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:16:27.522996 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.522981 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba635564-2081-4c62-9fb0-f96d16167263-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba635564-2081-4c62-9fb0-f96d16167263" (UID: "ba635564-2081-4c62-9fb0-f96d16167263"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:16:27.622192 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.622166 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.622192 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.622190 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvwkd\" (UniqueName: \"kubernetes.io/projected/ba635564-2081-4c62-9fb0-f96d16167263-kube-api-access-vvwkd\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.622334 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.622204 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba635564-2081-4c62-9fb0-f96d16167263-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:27.622334 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:27.622216 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba635564-2081-4c62-9fb0-f96d16167263-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:16:28.200814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.200784 2579 generic.go:356] "Generic (PLEG): container finished" podID="ba635564-2081-4c62-9fb0-f96d16167263" containerID="14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087" exitCode=0 Sep 11 14:16:28.200982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.200829 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" Sep 11 14:16:28.200982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.200832 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" event={"ID":"ba635564-2081-4c62-9fb0-f96d16167263","Type":"ContainerDied","Data":"14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087"} Sep 11 14:16:28.200982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.200873 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl" event={"ID":"ba635564-2081-4c62-9fb0-f96d16167263","Type":"ContainerDied","Data":"6118e6fed265efa8d6a0ee40ff3f3b29b9f838422f5e9c85410bb2aa19c02a9e"} Sep 11 14:16:28.200982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.200898 2579 scope.go:117] "RemoveContainer" containerID="14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087" Sep 11 14:16:28.210838 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.210820 2579 scope.go:117] "RemoveContainer" containerID="77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9" Sep 11 14:16:28.220124 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.220083 2579 scope.go:117] "RemoveContainer" containerID="14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087" Sep 11 14:16:28.220398 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:16:28.220380 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087\": container with ID starting with 14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087 not found: ID does not exist" containerID="14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087" Sep 11 14:16:28.220484 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.220408 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087"} err="failed to get container status \"14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087\": rpc error: code = NotFound desc = could not find container \"14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087\": container with ID starting with 14f199cbf816cb807ee16582729aaa02b04b99d2be336afc73e64528a4f3d087 not found: ID does not exist" Sep 11 14:16:28.220484 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.220429 2579 scope.go:117] "RemoveContainer" containerID="77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9" Sep 11 14:16:28.220704 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:16:28.220689 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9\": container with ID starting with 77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9 not found: ID does not exist" containerID="77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9" Sep 11 14:16:28.220752 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:28.220707 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9"} err="failed to get container status \"77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9\": rpc error: code = NotFound desc = could not find container \"77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9\": container with ID starting with 77214cf2ce70381b825a538ca7dd2a0ef4ac0b524baa6235a5138f8d2393d7d9 not found: ID does not exist" Sep 11 14:16:32.762337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:32.762299 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl"] Sep 11 14:16:35.471575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:35.471544 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d630fb00-openshift-default-8f6b98456-vrcjl"] Sep 11 14:16:35.643871 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:35.643837 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba635564-2081-4c62-9fb0-f96d16167263" path="/var/lib/kubelet/pods/ba635564-2081-4c62-9fb0-f96d16167263/volumes" Sep 11 14:16:37.539542 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:37.539499 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:51.055906 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:51.055871 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:51.137301 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:51.137268 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.394650 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:58.394610 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.581838 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:58.581808 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:16:58.725036 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:16:58.724953 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:03.907129 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:03.907085 2579 scope.go:117] "RemoveContainer" containerID="1530f62546b9c397f548f73b3d8f93456283dfedc5058202f65fb410a64746e0" Sep 11 14:17:09.143226 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:09.143189 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:09.370808 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:09.370768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:14.985625 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:14.985587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:30.280308 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:30.280275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:35.332583 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:35.332545 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:37.455413 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:37.455373 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:37.579329 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:37.579299 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:37.797502 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:37.797444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:37.923813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:37.923784 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:38.135917 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.135845 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88"] Sep 11 14:17:38.136369 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136354 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136371 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136383 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd3db8b-37ee-4a7a-9d77-215d0ff669ce" containerName="cleanup" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136389 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd3db8b-37ee-4a7a-9d77-215d0ff669ce" containerName="cleanup" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136399 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec718bd0-4d61-4e65-9440-d31ff3db3d96" containerName="cleanup" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136405 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec718bd0-4d61-4e65-9440-d31ff3db3d96" containerName="cleanup" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136415 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136420 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" Sep 11 14:17:38.136432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136433 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d70d6c-5209-43e0-a363-7f18858d7bfb" containerName="cleanup" Sep 11 14:17:38.136722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136438 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d70d6c-5209-43e0-a363-7f18858d7bfb" containerName="cleanup" Sep 11 14:17:38.136722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136515 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" Sep 11 14:17:38.136722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136530 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec718bd0-4d61-4e65-9440-d31ff3db3d96" containerName="cleanup" Sep 11 14:17:38.136722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136537 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d70d6c-5209-43e0-a363-7f18858d7bfb" containerName="cleanup" Sep 11 14:17:38.136722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136543 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba635564-2081-4c62-9fb0-f96d16167263" containerName="istio-proxy" Sep 11 14:17:38.136722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.136550 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd3db8b-37ee-4a7a-9d77-215d0ff669ce" containerName="cleanup" Sep 11 14:17:38.139731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.139715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.143357 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.143330 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-17d457-openshift-default-dockercfg-2r6h7\"" Sep 11 14:17:38.158539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.158513 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88"] Sep 11 14:17:38.180762 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.180732 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:38.265275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265236 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-envoy\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-credential-socket\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-data\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-podinfo\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-socket\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265449 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istiod-ca-cert\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rqnl\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-kube-api-access-2rqnl\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-certs\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.265746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.265638 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-token\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.332308 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.332282 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:38.367161 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-token\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367179 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-envoy\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-credential-socket\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-data\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367252 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-podinfo\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-socket\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istiod-ca-cert\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367679 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rqnl\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-kube-api-access-2rqnl\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367679 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-certs\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367782 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367679 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-credential-socket\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367782 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367734 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-data\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367858 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-socket\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.367992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.367957 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-certs\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.368130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.368073 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istiod-ca-cert\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.370004 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.369985 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-envoy\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.370262 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.370242 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-podinfo\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.381983 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.381965 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-token\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.382303 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.382285 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rqnl\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-kube-api-access-2rqnl\") pod \"e2e-api-a-17d457-openshift-default-f97bdc56-h7f88\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.450745 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.450691 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:38.741430 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:17:38.741403 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda938e6d4_5b4b_4189_88dd_28ea5cc43bdf.slice/crio-c0fcc8953fec92fe891f593346fc21bb47073ea6d7597e1855b9be68ac62e7d1 WatchSource:0}: Error finding container c0fcc8953fec92fe891f593346fc21bb47073ea6d7597e1855b9be68ac62e7d1: Status 404 returned error can't find the container with id c0fcc8953fec92fe891f593346fc21bb47073ea6d7597e1855b9be68ac62e7d1 Sep 11 14:17:38.741738 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.741493 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88"] Sep 11 14:17:38.743576 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.743547 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:38.743639 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.743610 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:38.743676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:38.743638 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:17:39.655458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:39.655424 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" event={"ID":"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf","Type":"ContainerStarted","Data":"397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba"} Sep 11 14:17:39.655643 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:39.655474 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" event={"ID":"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf","Type":"ContainerStarted","Data":"c0fcc8953fec92fe891f593346fc21bb47073ea6d7597e1855b9be68ac62e7d1"} Sep 11 14:17:39.685252 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:39.685204 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podStartSLOduration=1.685188173 podStartE2EDuration="1.685188173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:39.683160069 +0000 UTC m=+1476.552891356" watchObservedRunningTime="2026-09-11 14:17:39.685188173 +0000 UTC m=+1476.554919435" Sep 11 14:17:40.451584 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:40.451546 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:40.456814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:40.456792 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:40.662033 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:40.662003 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:40.662812 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:40.662793 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:17:41.257864 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:41.257825 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:42.073822 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:42.073783 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:42.495363 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:42.495290 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:44.005403 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:44.005366 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:45.792846 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:45.792813 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:47.564697 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:47.564660 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:48.780247 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:48.780196 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:50.037899 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:50.037861 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:50.151285 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:50.151250 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:50.820649 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:50.820612 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:50.928898 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:50.928866 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:17:51.122576 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.122506 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l"] Sep 11 14:17:51.126371 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.126353 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.129873 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.129852 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-17d457\"" Sep 11 14:17:51.130012 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.129891 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-17d457-dockercfg-8lr8l\"" Sep 11 14:17:51.139348 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.139328 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l"] Sep 11 14:17:51.289214 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.289173 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdqbb\" (UniqueName: \"kubernetes.io/projected/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-kube-api-access-bdqbb\") pod \"payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.289355 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.289248 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.390524 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.390453 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdqbb\" (UniqueName: \"kubernetes.io/projected/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-kube-api-access-bdqbb\") pod \"payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.390524 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.390500 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.391203 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.391185 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-plugins-config-volume\") pod \"payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.406181 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.406157 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdqbb\" (UniqueName: \"kubernetes.io/projected/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-kube-api-access-bdqbb\") pod \"payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.438224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.438202 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:51.725878 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.725850 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l"] Sep 11 14:17:51.726786 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:17:51.726766 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bd7f1d_7b4b_4a45_a22e_b93cee9bf090.slice/crio-2851d95182fc50a57cb929995e4442c86f8283a59c563ffc98b763872689f940 WatchSource:0}: Error finding container 2851d95182fc50a57cb929995e4442c86f8283a59c563ffc98b763872689f940: Status 404 returned error can't find the container with id 2851d95182fc50a57cb929995e4442c86f8283a59c563ffc98b763872689f940 Sep 11 14:17:51.741630 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:51.741605 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" event={"ID":"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090","Type":"ContainerStarted","Data":"2851d95182fc50a57cb929995e4442c86f8283a59c563ffc98b763872689f940"} Sep 11 14:17:52.749054 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:52.749019 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" event={"ID":"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090","Type":"ContainerStarted","Data":"b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013"} Sep 11 14:17:52.749492 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:52.749085 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:17:52.781897 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:52.781847 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" podStartSLOduration=1.781831134 podStartE2EDuration="1.781831134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:17:52.780809533 +0000 UTC m=+1489.650540795" watchObservedRunningTime="2026-09-11 14:17:52.781831134 +0000 UTC m=+1489.651562396" Sep 11 14:17:59.672519 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:17:59.672473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:03.757037 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:03.757011 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:18:05.126862 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:05.126826 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:07.787343 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:07.787304 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:08.718621 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.718584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:08.862392 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.862320 2579 generic.go:356] "Generic (PLEG): container finished" podID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerID="397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba" exitCode=0 Sep 11 14:18:08.862797 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.862394 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" event={"ID":"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf","Type":"ContainerDied","Data":"397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba"} Sep 11 14:18:08.862797 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.862783 2579 scope.go:117] "RemoveContainer" containerID="397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba" Sep 11 14:18:08.863937 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.863911 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:08.863993 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.863979 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:08.864035 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:08.864010 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:09.872355 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:09.872315 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" event={"ID":"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf","Type":"ContainerStarted","Data":"5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74"} Sep 11 14:18:10.451298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:10.451263 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:18:10.452200 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:10.452177 2579 patch_prober.go:28] interesting pod/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:18:10.452274 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:10.452224 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 11 14:18:10.662119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:10.662082 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:18:11.451707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:11.451675 2579 patch_prober.go:28] interesting pod/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:18:11.452128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:11.451737 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 11 14:18:11.460803 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:11.460773 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:12.451402 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:12.451370 2579 patch_prober.go:28] interesting pod/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:18:12.451596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:12.451422 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 11 14:18:13.132274 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:13.132238 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:13.341331 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:13.341278 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:13.451987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:13.451886 2579 patch_prober.go:28] interesting pod/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:18:13.451987 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:13.451954 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 11 14:18:14.431702 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:14.431663 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:14.451383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:14.451353 2579 patch_prober.go:28] interesting pod/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Sep 11 14:18:14.451503 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:14.451399 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Sep 11 14:18:15.283682 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:15.283646 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:15.455307 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:15.455275 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:18:15.456238 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:15.456218 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:18:16.160129 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:16.160084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:16.915375 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:16.915342 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:17.301112 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.301065 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:17.474455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.474421 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:17.848333 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.848300 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4"] Sep 11 14:18:17.851413 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.851397 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.854291 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.854271 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-106468-openshift-default-dockercfg-7hgqs\"" Sep 11 14:18:17.855876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.855847 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:17.885167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.885144 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4"] Sep 11 14:18:17.958861 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.958824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istiod-ca-cert\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.958861 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.958859 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-credential-socket\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.958878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-data\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.958896 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-certs\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.958921 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-envoy\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.959013 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-podinfo\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.959063 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-token\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.959139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwdj\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-kube-api-access-bfwdj\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:17.959393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:17.959184 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-socket\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.059927 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.059895 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwdj\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-kube-api-access-bfwdj\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.059943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-socket\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istiod-ca-cert\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060062 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-credential-socket\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060091 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-data\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060150 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-certs\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-envoy\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-podinfo\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060288 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-token\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060579 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060469 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-socket\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060579 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060504 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-credential-socket\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060697 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060674 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-data\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060738 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060699 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-certs\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.060810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.060792 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istiod-ca-cert\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.062719 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.062697 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-envoy\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.062979 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.062960 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-podinfo\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.069509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.069488 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwdj\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-kube-api-access-bfwdj\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.074173 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.074154 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-token\") pod \"e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.163294 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.163236 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:18.392604 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.392568 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:18.456504 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.456475 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4"] Sep 11 14:18:18.457288 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:18:18.457259 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ca7778f_a232_4ff6_ab16_423c9dd7fcef.slice/crio-2968c1a0c751eb05c9980f460900d39f5abb25d0bae6f9fa363395e7c44ad4fc WatchSource:0}: Error finding container 2968c1a0c751eb05c9980f460900d39f5abb25d0bae6f9fa363395e7c44ad4fc: Status 404 returned error can't find the container with id 2968c1a0c751eb05c9980f460900d39f5abb25d0bae6f9fa363395e7c44ad4fc Sep 11 14:18:18.459537 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.459507 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:18.459611 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.459579 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:18.459651 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.459609 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:18.527294 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.527266 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:18.959312 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.959279 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" event={"ID":"0ca7778f-a232-4ff6-ab16-423c9dd7fcef","Type":"ContainerStarted","Data":"9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37"} Sep 11 14:18:18.959713 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.959330 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" event={"ID":"0ca7778f-a232-4ff6-ab16-423c9dd7fcef","Type":"ContainerStarted","Data":"2968c1a0c751eb05c9980f460900d39f5abb25d0bae6f9fa363395e7c44ad4fc"} Sep 11 14:18:18.993754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:18.993704 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podStartSLOduration=1.9936864650000001 podStartE2EDuration="1.993686465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:18.991030239 +0000 UTC m=+1515.860761512" watchObservedRunningTime="2026-09-11 14:18:18.993686465 +0000 UTC m=+1515.863417727" Sep 11 14:18:19.164303 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:19.164273 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:19.169527 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:19.169505 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:19.786065 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:19.786031 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:19.965807 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:19.965776 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:19.966717 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:19.966698 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:21.983627 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:21.983589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:22.166520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:22.166484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:22.470283 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:22.470247 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:23.836087 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:23.836048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:24.584355 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:24.584329 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:25.393533 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:25.393501 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:26.336940 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:26.336881 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:27.623910 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:27.623876 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:27.784777 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:27.784739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:27.905553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:27.905459 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:28.434989 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:28.434950 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:28.533167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:28.533135 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:40.981159 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:40.981120 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:44.163315 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:44.163276 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerID="9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37" exitCode=0 Sep 11 14:18:44.163750 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:44.163353 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" event={"ID":"0ca7778f-a232-4ff6-ab16-423c9dd7fcef","Type":"ContainerDied","Data":"9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37"} Sep 11 14:18:44.163822 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:44.163806 2579 scope.go:117] "RemoveContainer" containerID="9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37" Sep 11 14:18:44.165055 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:44.165028 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:44.165151 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:44.165133 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:44.165202 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:44.165168 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:18:45.173450 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:45.173413 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" event={"ID":"0ca7778f-a232-4ff6-ab16-423c9dd7fcef","Type":"ContainerStarted","Data":"d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8"} Sep 11 14:18:46.163530 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:46.163487 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:46.164841 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:46.164815 2579 patch_prober.go:28] interesting pod/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:18:46.164963 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:46.164863 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 11 14:18:47.163725 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:47.163691 2579 patch_prober.go:28] interesting pod/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:18:47.164138 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:47.163747 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 11 14:18:48.164298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:48.164262 2579 patch_prober.go:28] interesting pod/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:18:48.164711 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:48.164351 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 11 14:18:49.164244 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:49.164204 2579 patch_prober.go:28] interesting pod/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:18:49.164432 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:49.164263 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 11 14:18:49.965979 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:49.965939 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:50.164839 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:50.164803 2579 patch_prober.go:28] interesting pod/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Sep 11 14:18:50.165235 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:50.164863 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Sep 11 14:18:50.511013 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:50.510978 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:51.100221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:51.100180 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:51.168490 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:51.168454 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:51.169557 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:51.169534 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:18:52.047191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:52.047150 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:53.085509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:53.085451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:53.892393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:53.892356 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:53.989285 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:53.989251 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f"] Sep 11 14:18:53.992191 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:53.992167 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:53.995828 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:53.995801 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:18:53.995973 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:53.995865 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:18:54.003626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.003602 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f"] Sep 11 14:18:54.051603 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.051568 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca10ea-af8d-45ea-98bb-b55a8049bdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d457-qv69f\" (UID: \"09ca10ea-af8d-45ea-98bb-b55a8049bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:54.153082 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.152989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca10ea-af8d-45ea-98bb-b55a8049bdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d457-qv69f\" (UID: \"09ca10ea-af8d-45ea-98bb-b55a8049bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:54.153646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.153627 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca10ea-af8d-45ea-98bb-b55a8049bdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-17d457-qv69f\" (UID: \"09ca10ea-af8d-45ea-98bb-b55a8049bdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:54.305086 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.305053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:54.592070 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.592040 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f"] Sep 11 14:18:54.593763 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:18:54.593734 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ca10ea_af8d_45ea_98bb_b55a8049bdc4.slice/crio-efe5585a3dba3c25f17b0862e0c361119b0583de1ad85d5e374f42c1e2c247f0 WatchSource:0}: Error finding container efe5585a3dba3c25f17b0862e0c361119b0583de1ad85d5e374f42c1e2c247f0: Status 404 returned error can't find the container with id efe5585a3dba3c25f17b0862e0c361119b0583de1ad85d5e374f42c1e2c247f0 Sep 11 14:18:54.799364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:54.799336 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:55.247455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:55.247365 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" event={"ID":"09ca10ea-af8d-45ea-98bb-b55a8049bdc4","Type":"ContainerStarted","Data":"2cd6a8f9c813cfa3f96a4f4703a8b43980e9ddc4b55d0833bfc38d9fbac1482c"} Sep 11 14:18:55.247455 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:55.247446 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" event={"ID":"09ca10ea-af8d-45ea-98bb-b55a8049bdc4","Type":"ContainerStarted","Data":"efe5585a3dba3c25f17b0862e0c361119b0583de1ad85d5e374f42c1e2c247f0"} Sep 11 14:18:55.267449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:55.267376 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" podStartSLOduration=2.267362845 podStartE2EDuration="2.267362845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:18:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:18:55.265844994 +0000 UTC m=+1552.135576272" watchObservedRunningTime="2026-09-11 14:18:55.267362845 +0000 UTC m=+1552.137094107" Sep 11 14:18:55.769551 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:55.769520 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:56.256011 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:56.255971 2579 generic.go:356] "Generic (PLEG): container finished" podID="09ca10ea-af8d-45ea-98bb-b55a8049bdc4" containerID="2cd6a8f9c813cfa3f96a4f4703a8b43980e9ddc4b55d0833bfc38d9fbac1482c" exitCode=0 Sep 11 14:18:56.256446 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:56.256026 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" event={"ID":"09ca10ea-af8d-45ea-98bb-b55a8049bdc4","Type":"ContainerDied","Data":"2cd6a8f9c813cfa3f96a4f4703a8b43980e9ddc4b55d0833bfc38d9fbac1482c"} Sep 11 14:18:56.859855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:56.859823 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:57.411367 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:57.411338 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:57.490048 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:57.490016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca10ea-af8d-45ea-98bb-b55a8049bdc4-maas-api-service-ca\") pod \"09ca10ea-af8d-45ea-98bb-b55a8049bdc4\" (UID: \"09ca10ea-af8d-45ea-98bb-b55a8049bdc4\") " Sep 11 14:18:57.490441 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:57.490423 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ca10ea-af8d-45ea-98bb-b55a8049bdc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09ca10ea-af8d-45ea-98bb-b55a8049bdc4" (UID: "09ca10ea-af8d-45ea-98bb-b55a8049bdc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:18:57.591726 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:57.591693 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09ca10ea-af8d-45ea-98bb-b55a8049bdc4-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:18:57.820948 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:57.820908 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:18:58.270664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:58.270630 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" Sep 11 14:18:58.270664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:58.270658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f" event={"ID":"09ca10ea-af8d-45ea-98bb-b55a8049bdc4","Type":"ContainerDied","Data":"efe5585a3dba3c25f17b0862e0c361119b0583de1ad85d5e374f42c1e2c247f0"} Sep 11 14:18:58.270878 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:58.270709 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efe5585a3dba3c25f17b0862e0c361119b0583de1ad85d5e374f42c1e2c247f0" Sep 11 14:18:59.353681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:18:59.353647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:00.287394 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:00.287343 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:01.372835 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:01.372790 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:04.156707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:04.156670 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:10.082646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.082610 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l"] Sep 11 14:19:10.083343 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.082817 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" podUID="e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" containerName="payload-processing" containerID="cri-o://b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013" gracePeriod=30 Sep 11 14:19:10.144709 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.144675 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:10.352168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.352141 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:19:10.360536 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.360494 2579 generic.go:356] "Generic (PLEG): container finished" podID="e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" containerID="b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013" exitCode=0 Sep 11 14:19:10.361141 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.361080 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" event={"ID":"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090","Type":"ContainerDied","Data":"b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013"} Sep 11 14:19:10.361440 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.361399 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" event={"ID":"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090","Type":"ContainerDied","Data":"2851d95182fc50a57cb929995e4442c86f8283a59c563ffc98b763872689f940"} Sep 11 14:19:10.379824 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.377746 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l" Sep 11 14:19:10.379824 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.377769 2579 scope.go:117] "RemoveContainer" containerID="b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013" Sep 11 14:19:10.400472 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.400165 2579 scope.go:117] "RemoveContainer" containerID="b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013" Sep 11 14:19:10.400858 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:19:10.400759 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013\": container with ID starting with b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013 not found: ID does not exist" containerID="b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013" Sep 11 14:19:10.400858 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.400801 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013"} err="failed to get container status \"b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013\": rpc error: code = NotFound desc = could not find container \"b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013\": container with ID starting with b07961a8504d64671b6495479891d9bcdf99b215628fc4d699f2146b61b82013 not found: ID does not exist" Sep 11 14:19:10.421165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.421130 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdqbb\" (UniqueName: \"kubernetes.io/projected/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-kube-api-access-bdqbb\") pod \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " Sep 11 14:19:10.421359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.421251 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-plugins-config-volume\") pod \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\" (UID: \"e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090\") " Sep 11 14:19:10.421614 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.421588 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" (UID: "e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:10.423539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.423512 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-kube-api-access-bdqbb" (OuterVolumeSpecName: "kube-api-access-bdqbb") pod "e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" (UID: "e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090"). InnerVolumeSpecName "kube-api-access-bdqbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:10.522891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.522854 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdqbb\" (UniqueName: \"kubernetes.io/projected/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-kube-api-access-bdqbb\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.522891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.522886 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:10.550846 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.550815 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f"] Sep 11 14:19:10.557720 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.557696 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-17d457-qv69f"] Sep 11 14:19:10.717165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.717132 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l"] Sep 11 14:19:10.724790 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:10.724762 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-17d457-7cbb86c9d7-xpc8l"] Sep 11 14:19:11.643954 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:11.643919 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09ca10ea-af8d-45ea-98bb-b55a8049bdc4" path="/var/lib/kubelet/pods/09ca10ea-af8d-45ea-98bb-b55a8049bdc4/volumes" Sep 11 14:19:11.644357 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:11.644327 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" path="/var/lib/kubelet/pods/e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090/volumes" Sep 11 14:19:11.837662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:11.837589 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:12.070228 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:12.070191 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:13.097154 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:13.097093 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:14.038151 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:14.038079 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:14.836150 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:14.836082 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:16.007150 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:16.007116 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:17.603582 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:17.603548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:17.715850 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:17.715810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:17.910549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:17.910463 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88"] Sep 11 14:19:17.910730 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:17.910707 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" containerID="cri-o://5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74" gracePeriod=30 Sep 11 14:19:17.951167 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:17.951131 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:18.254218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254132 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c"] Sep 11 14:19:18.254610 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254597 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" containerName="payload-processing" Sep 11 14:19:18.254657 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254611 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" containerName="payload-processing" Sep 11 14:19:18.254657 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254620 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09ca10ea-af8d-45ea-98bb-b55a8049bdc4" containerName="revoke-keys" Sep 11 14:19:18.254657 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254625 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="09ca10ea-af8d-45ea-98bb-b55a8049bdc4" containerName="revoke-keys" Sep 11 14:19:18.254776 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254706 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bd7f1d-7b4b-4a45-a22e-b93cee9bf090" containerName="payload-processing" Sep 11 14:19:18.254776 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.254718 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="09ca10ea-af8d-45ea-98bb-b55a8049bdc4" containerName="revoke-keys" Sep 11 14:19:18.257821 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.257805 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:18.261701 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.261667 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:19:18.261844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.261710 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:19:18.271509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.271478 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c"] Sep 11 14:19:18.407795 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.407749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:18.407795 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.407781 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8726de38-dd1b-4185-ad22-5f063f981711-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-106468-4ps2c\" (UID: \"8726de38-dd1b-4185-ad22-5f063f981711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:18.508688 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.508598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8726de38-dd1b-4185-ad22-5f063f981711-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-106468-4ps2c\" (UID: \"8726de38-dd1b-4185-ad22-5f063f981711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:18.509255 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.509236 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8726de38-dd1b-4185-ad22-5f063f981711-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-106468-4ps2c\" (UID: \"8726de38-dd1b-4185-ad22-5f063f981711\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:18.569230 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.569197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:18.862664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:18.862640 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c"] Sep 11 14:19:18.864238 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:19:18.864211 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8726de38_dd1b_4185_ad22_5f063f981711.slice/crio-05837d43fcd6efb0c930aabd4cf0730d5eb5366555cf5ee4d5a162e2ed802bef WatchSource:0}: Error finding container 05837d43fcd6efb0c930aabd4cf0730d5eb5366555cf5ee4d5a162e2ed802bef: Status 404 returned error can't find the container with id 05837d43fcd6efb0c930aabd4cf0730d5eb5366555cf5ee4d5a162e2ed802bef Sep 11 14:19:19.429693 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:19.429610 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" event={"ID":"8726de38-dd1b-4185-ad22-5f063f981711","Type":"ContainerStarted","Data":"2c53799adb955526d02f58e19cdb9db80a060655ddefbce292c4e609e55d6ffe"} Sep 11 14:19:19.429693 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:19.429658 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" event={"ID":"8726de38-dd1b-4185-ad22-5f063f981711","Type":"ContainerStarted","Data":"05837d43fcd6efb0c930aabd4cf0730d5eb5366555cf5ee4d5a162e2ed802bef"} Sep 11 14:19:19.448463 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:19.448434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:19.459993 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:19.459795 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" podStartSLOduration=1.459773616 podStartE2EDuration="1.459773616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:19.458227211 +0000 UTC m=+1576.327958474" watchObservedRunningTime="2026-09-11 14:19:19.459773616 +0000 UTC m=+1576.329504880" Sep 11 14:19:20.438281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:20.438243 2579 generic.go:356] "Generic (PLEG): container finished" podID="8726de38-dd1b-4185-ad22-5f063f981711" containerID="2c53799adb955526d02f58e19cdb9db80a060655ddefbce292c4e609e55d6ffe" exitCode=0 Sep 11 14:19:20.438281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:20.438286 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" event={"ID":"8726de38-dd1b-4185-ad22-5f063f981711","Type":"ContainerDied","Data":"2c53799adb955526d02f58e19cdb9db80a060655ddefbce292c4e609e55d6ffe"} Sep 11 14:19:20.587887 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:20.587854 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:21.054984 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:21.054949 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:21.516782 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:21.516754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:21.604985 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:21.604961 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:21.740089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:21.739996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8726de38-dd1b-4185-ad22-5f063f981711-maas-api-service-ca\") pod \"8726de38-dd1b-4185-ad22-5f063f981711\" (UID: \"8726de38-dd1b-4185-ad22-5f063f981711\") " Sep 11 14:19:21.740433 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:21.740413 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8726de38-dd1b-4185-ad22-5f063f981711-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8726de38-dd1b-4185-ad22-5f063f981711" (UID: "8726de38-dd1b-4185-ad22-5f063f981711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:21.841592 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:21.841549 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8726de38-dd1b-4185-ad22-5f063f981711-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:22.453125 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:22.453072 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" Sep 11 14:19:22.453312 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:22.453125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c" event={"ID":"8726de38-dd1b-4185-ad22-5f063f981711","Type":"ContainerDied","Data":"05837d43fcd6efb0c930aabd4cf0730d5eb5366555cf5ee4d5a162e2ed802bef"} Sep 11 14:19:22.453312 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:22.453168 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05837d43fcd6efb0c930aabd4cf0730d5eb5366555cf5ee4d5a162e2ed802bef" Sep 11 14:19:22.598123 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:22.598066 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:23.173325 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.173252 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:19:23.253916 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.253882 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-data\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.253938 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rqnl\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-kube-api-access-2rqnl\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.253979 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istiod-ca-cert\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.253992 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-token\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254012 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-envoy\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254034 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-credential-socket\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254059 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-certs\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254094 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-podinfo\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254250 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-socket\") pod \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\" (UID: \"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf\") " Sep 11 14:19:23.254510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254285 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-data" (OuterVolumeSpecName: "istio-data") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:23.254510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254364 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:23.254510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254409 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:23.254742 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254533 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:23.254742 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254655 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:23.254742 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254731 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.254904 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254751 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.254904 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254766 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.254904 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.254797 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.256576 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.256547 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-kube-api-access-2rqnl" (OuterVolumeSpecName: "kube-api-access-2rqnl") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "kube-api-access-2rqnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:23.256685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.256574 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:23.256685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.256592 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-token" (OuterVolumeSpecName: "istio-token") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:23.256685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.256602 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" (UID: "a938e6d4-5b4b-4189-88dd-28ea5cc43bdf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:19:23.355499 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.355467 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rqnl\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-kube-api-access-2rqnl\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.355499 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.355493 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.355499 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.355503 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.355731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.355511 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.355731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.355519 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:23.377634 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.377604 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:23.461862 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.461782 2579 generic.go:356] "Generic (PLEG): container finished" podID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerID="5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74" exitCode=0 Sep 11 14:19:23.461862 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.461837 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" Sep 11 14:19:23.462041 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.461862 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" event={"ID":"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf","Type":"ContainerDied","Data":"5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74"} Sep 11 14:19:23.462041 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.461953 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88" event={"ID":"a938e6d4-5b4b-4189-88dd-28ea5cc43bdf","Type":"ContainerDied","Data":"c0fcc8953fec92fe891f593346fc21bb47073ea6d7597e1855b9be68ac62e7d1"} Sep 11 14:19:23.462041 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.461979 2579 scope.go:117] "RemoveContainer" containerID="5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74" Sep 11 14:19:23.472824 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.472803 2579 scope.go:117] "RemoveContainer" containerID="397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba" Sep 11 14:19:23.482529 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.482512 2579 scope.go:117] "RemoveContainer" containerID="5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74" Sep 11 14:19:23.482793 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:19:23.482777 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74\": container with ID starting with 5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74 not found: ID does not exist" containerID="5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74" Sep 11 14:19:23.482847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.482800 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74"} err="failed to get container status \"5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74\": rpc error: code = NotFound desc = could not find container \"5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74\": container with ID starting with 5b6ef91547cce22c698718b5a1f4682af61b3120fa4d69ada46f49cc77bfee74 not found: ID does not exist" Sep 11 14:19:23.482847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.482817 2579 scope.go:117] "RemoveContainer" containerID="397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba" Sep 11 14:19:23.483062 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:19:23.483046 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba\": container with ID starting with 397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba not found: ID does not exist" containerID="397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba" Sep 11 14:19:23.483124 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.483068 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba"} err="failed to get container status \"397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba\": rpc error: code = NotFound desc = could not find container \"397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba\": container with ID starting with 397f29f14f2bdd3bbd53e58884b580f5efc5c06ff86bbcf0527406d58bd01cba not found: ID does not exist" Sep 11 14:19:23.488535 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.488503 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88"] Sep 11 14:19:23.494779 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.494757 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-17d457-openshift-default-f97bdc56-h7f88"] Sep 11 14:19:23.643902 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:23.643869 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" path="/var/lib/kubelet/pods/a938e6d4-5b4b-4189-88dd-28ea5cc43bdf/volumes" Sep 11 14:19:24.144283 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:24.144253 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:24.244745 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:24.244709 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:34.249046 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:34.249008 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:34.387565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:34.387530 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:34.530217 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:34.530081 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:34.627524 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:34.627491 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c"] Sep 11 14:19:34.631950 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:34.631920 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-106468-4ps2c"] Sep 11 14:19:34.635782 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:34.635753 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:35.045924 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:35.045893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:35.153672 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:35.153640 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:35.643253 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:35.643221 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8726de38-dd1b-4185-ad22-5f063f981711" path="/var/lib/kubelet/pods/8726de38-dd1b-4185-ad22-5f063f981711/volumes" Sep 11 14:19:35.991371 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:35.991279 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:37.240274 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:37.240244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:37.909793 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:37.909755 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:38.495792 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:38.495754 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:38.957594 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:38.957555 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:39.366504 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.366473 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:39.470356 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.470318 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:39.882282 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882246 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb"] Sep 11 14:19:39.882719 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882706 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" Sep 11 14:19:39.882764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882721 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" Sep 11 14:19:39.882764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882743 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8726de38-dd1b-4185-ad22-5f063f981711" containerName="revoke-keys" Sep 11 14:19:39.882764 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882751 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="8726de38-dd1b-4185-ad22-5f063f981711" containerName="revoke-keys" Sep 11 14:19:39.882855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882836 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" Sep 11 14:19:39.882855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882845 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="8726de38-dd1b-4185-ad22-5f063f981711" containerName="revoke-keys" Sep 11 14:19:39.882855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.882850 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" Sep 11 14:19:39.886290 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.886266 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:39.893531 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.893318 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e6521d-openshift-default-dockercfg-56cb4\"" Sep 11 14:19:39.919867 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.919836 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb"] Sep 11 14:19:39.982198 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:39.982139 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:40.029075 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029046 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-token\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029131 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-socket\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029221 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z56ll\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-kube-api-access-z56ll\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-credential-socket\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029294 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-data\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029313 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istiod-ca-cert\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029440 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029344 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-envoy\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029440 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029391 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-certs\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.029503 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.029447 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-podinfo\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130406 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-podinfo\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130428 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-token\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-socket\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130498 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z56ll\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-kube-api-access-z56ll\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130538 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-credential-socket\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130560 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-data\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130581 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130579 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istiod-ca-cert\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130932 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130602 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-envoy\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.130932 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130635 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-certs\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.131042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130937 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-socket\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.131042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.130989 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-credential-socket\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.131042 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.131017 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-data\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.131201 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.131052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-certs\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.131428 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.131409 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istiod-ca-cert\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.133126 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.133050 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-envoy\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.133369 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.133340 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-podinfo\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.145950 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.145919 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z56ll\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-kube-api-access-z56ll\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.147021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.146998 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-token\") pod \"e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.166866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.166835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:40.196961 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.196929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:40.485336 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.485307 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb"] Sep 11 14:19:40.485999 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:19:40.485976 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac1b6f52_816f_4d7f_8840_0a76bbccf1b1.slice/crio-408bb8252065afabb1d0eb867c6b259df69cde798e002132b047ce64302c004d WatchSource:0}: Error finding container 408bb8252065afabb1d0eb867c6b259df69cde798e002132b047ce64302c004d: Status 404 returned error can't find the container with id 408bb8252065afabb1d0eb867c6b259df69cde798e002132b047ce64302c004d Sep 11 14:19:40.488212 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.488182 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:40.488292 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.488242 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:40.488292 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.488273 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:19:40.576396 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.576364 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" event={"ID":"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1","Type":"ContainerStarted","Data":"33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68"} Sep 11 14:19:40.576579 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.576422 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" event={"ID":"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1","Type":"ContainerStarted","Data":"408bb8252065afabb1d0eb867c6b259df69cde798e002132b047ce64302c004d"} Sep 11 14:19:40.613231 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:40.613168 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podStartSLOduration=1.6131451220000002 podStartE2EDuration="1.613145122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:40.61028292 +0000 UTC m=+1597.480014182" watchObservedRunningTime="2026-09-11 14:19:40.613145122 +0000 UTC m=+1597.482876400" Sep 11 14:19:41.198038 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:41.197998 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:41.203409 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:41.203378 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:41.582792 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:41.582759 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:41.583786 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:41.583767 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:19:41.985710 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:41.985620 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:42.821216 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:42.821178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:42.943219 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:42.943178 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:43.269438 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:43.269358 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:43.756843 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:43.756810 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:43.970287 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:43.970248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:44.046757 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.046722 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4"] Sep 11 14:19:44.047159 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.047084 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" containerID="cri-o://d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8" gracePeriod=30 Sep 11 14:19:44.073501 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.073460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:44.385183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.385083 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx"] Sep 11 14:19:44.385600 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.385588 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" Sep 11 14:19:44.385694 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.385602 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="a938e6d4-5b4b-4189-88dd-28ea5cc43bdf" containerName="istio-proxy" Sep 11 14:19:44.389026 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.389008 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:44.392433 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.392411 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:19:44.392536 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.392455 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:19:44.401378 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.401351 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx"] Sep 11 14:19:44.479209 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.479176 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f3bdc3-0e8e-481d-bbe1-24c4362a0896-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx\" (UID: \"90f3bdc3-0e8e-481d-bbe1-24c4362a0896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:44.580132 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.580072 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f3bdc3-0e8e-481d-bbe1-24c4362a0896-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx\" (UID: \"90f3bdc3-0e8e-481d-bbe1-24c4362a0896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:44.580753 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.580733 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f3bdc3-0e8e-481d-bbe1-24c4362a0896-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx\" (UID: \"90f3bdc3-0e8e-481d-bbe1-24c4362a0896\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:44.701205 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.701125 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:44.803154 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.803089 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:44.986395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:44.986366 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx"] Sep 11 14:19:44.986820 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:19:44.986417 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f3bdc3_0e8e_481d_bbe1_24c4362a0896.slice/crio-b7bc666e77507e6e972287aac9474f6199418027f88ffc5245851b1889ca2c32 WatchSource:0}: Error finding container b7bc666e77507e6e972287aac9474f6199418027f88ffc5245851b1889ca2c32: Status 404 returned error can't find the container with id b7bc666e77507e6e972287aac9474f6199418027f88ffc5245851b1889ca2c32 Sep 11 14:19:45.610311 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:45.610277 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" event={"ID":"90f3bdc3-0e8e-481d-bbe1-24c4362a0896","Type":"ContainerStarted","Data":"032c479ac658ba54f2b4b9ba7a4b8a368eab66ec62b7e60d6bc0d5fd92276bbb"} Sep 11 14:19:45.610504 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:45.610331 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" event={"ID":"90f3bdc3-0e8e-481d-bbe1-24c4362a0896","Type":"ContainerStarted","Data":"b7bc666e77507e6e972287aac9474f6199418027f88ffc5245851b1889ca2c32"} Sep 11 14:19:45.633692 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:45.633639 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" podStartSLOduration=1.63362462 podStartE2EDuration="1.63362462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:45.632166838 +0000 UTC m=+1602.501898104" watchObservedRunningTime="2026-09-11 14:19:45.63362462 +0000 UTC m=+1602.503355883" Sep 11 14:19:46.310416 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:46.310374 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:46.618061 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:46.617968 2579 generic.go:356] "Generic (PLEG): container finished" podID="90f3bdc3-0e8e-481d-bbe1-24c4362a0896" containerID="032c479ac658ba54f2b4b9ba7a4b8a368eab66ec62b7e60d6bc0d5fd92276bbb" exitCode=0 Sep 11 14:19:46.618234 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:46.618059 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" event={"ID":"90f3bdc3-0e8e-481d-bbe1-24c4362a0896","Type":"ContainerDied","Data":"032c479ac658ba54f2b4b9ba7a4b8a368eab66ec62b7e60d6bc0d5fd92276bbb"} Sep 11 14:19:47.600086 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:47.600048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:47.774899 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:47.774876 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:47.915956 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:47.915859 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f3bdc3-0e8e-481d-bbe1-24c4362a0896-maas-api-service-ca\") pod \"90f3bdc3-0e8e-481d-bbe1-24c4362a0896\" (UID: \"90f3bdc3-0e8e-481d-bbe1-24c4362a0896\") " Sep 11 14:19:47.918214 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:47.916928 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f3bdc3-0e8e-481d-bbe1-24c4362a0896-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90f3bdc3-0e8e-481d-bbe1-24c4362a0896" (UID: "90f3bdc3-0e8e-481d-bbe1-24c4362a0896"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:48.017435 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.017401 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90f3bdc3-0e8e-481d-bbe1-24c4362a0896-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:48.640096 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.640056 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" event={"ID":"90f3bdc3-0e8e-481d-bbe1-24c4362a0896","Type":"ContainerDied","Data":"b7bc666e77507e6e972287aac9474f6199418027f88ffc5245851b1889ca2c32"} Sep 11 14:19:48.640605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.640067 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx" Sep 11 14:19:48.640605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.640135 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7bc666e77507e6e972287aac9474f6199418027f88ffc5245851b1889ca2c32" Sep 11 14:19:48.686960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.686927 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:48.860460 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.860416 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:48.977832 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:48.977740 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:49.325117 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.325075 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:19:49.357883 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.357838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:49.433029 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.432995 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-data\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433029 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433035 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-certs\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433067 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-socket\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433093 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istiod-ca-cert\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433225 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-token\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433275 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-envoy\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433315 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bfwdj\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-kube-api-access-bfwdj\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433593 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433341 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-podinfo\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433593 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433361 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:49.433593 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433400 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-credential-socket\") pod \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\" (UID: \"0ca7778f-a232-4ff6-ab16-423c9dd7fcef\") " Sep 11 14:19:49.433593 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433400 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:49.433593 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433413 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-data" (OuterVolumeSpecName: "istio-data") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:49.433876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433610 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:49.433876 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.433854 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:49.434218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.434122 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.434218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.434149 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.434218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.434162 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.434218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.434176 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.434218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.434190 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.435647 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.435624 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-token" (OuterVolumeSpecName: "istio-token") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:49.435757 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.435645 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:19:49.435757 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.435662 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-kube-api-access-bfwdj" (OuterVolumeSpecName: "kube-api-access-bfwdj") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "kube-api-access-bfwdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:49.435831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.435794 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0ca7778f-a232-4ff6-ab16-423c9dd7fcef" (UID: "0ca7778f-a232-4ff6-ab16-423c9dd7fcef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:19:49.461230 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.461202 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:49.490374 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490343 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk"] Sep 11 14:19:49.490800 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490787 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" Sep 11 14:19:49.490851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490801 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" Sep 11 14:19:49.490851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490812 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" Sep 11 14:19:49.490851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490818 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" Sep 11 14:19:49.490851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490823 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90f3bdc3-0e8e-481d-bbe1-24c4362a0896" containerName="revoke-keys" Sep 11 14:19:49.490851 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490828 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="90f3bdc3-0e8e-481d-bbe1-24c4362a0896" containerName="revoke-keys" Sep 11 14:19:49.491016 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490915 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" Sep 11 14:19:49.491016 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.490928 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="90f3bdc3-0e8e-481d-bbe1-24c4362a0896" containerName="revoke-keys" Sep 11 14:19:49.494509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.494492 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.498992 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.498974 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e6521d\"" Sep 11 14:19:49.510855 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.510833 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk"] Sep 11 14:19:49.535089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.535064 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.535089 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.535085 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.535304 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.535095 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwdj\" (UniqueName: \"kubernetes.io/projected/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-kube-api-access-bfwdj\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.535304 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.535117 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ca7778f-a232-4ff6-ab16-423c9dd7fcef-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:49.635949 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.635846 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4js8w\" (UniqueName: \"kubernetes.io/projected/68fe1636-5684-40a1-8a15-54685f37821c-kube-api-access-4js8w\") pod \"payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.636152 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.636032 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fe1636-5684-40a1-8a15-54685f37821c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.663888 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.655164 2579 generic.go:356] "Generic (PLEG): container finished" podID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerID="d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8" exitCode=0 Sep 11 14:19:49.663888 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.655321 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" event={"ID":"0ca7778f-a232-4ff6-ab16-423c9dd7fcef","Type":"ContainerDied","Data":"d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8"} Sep 11 14:19:49.663888 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.655402 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" event={"ID":"0ca7778f-a232-4ff6-ab16-423c9dd7fcef","Type":"ContainerDied","Data":"2968c1a0c751eb05c9980f460900d39f5abb25d0bae6f9fa363395e7c44ad4fc"} Sep 11 14:19:49.663888 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.655434 2579 scope.go:117] "RemoveContainer" containerID="d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8" Sep 11 14:19:49.663888 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.655671 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4" Sep 11 14:19:49.689944 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.689916 2579 scope.go:117] "RemoveContainer" containerID="9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37" Sep 11 14:19:49.699767 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.699711 2579 scope.go:117] "RemoveContainer" containerID="d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8" Sep 11 14:19:49.700227 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:19:49.700196 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8\": container with ID starting with d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8 not found: ID does not exist" containerID="d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8" Sep 11 14:19:49.700431 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.700385 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8"} err="failed to get container status \"d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8\": rpc error: code = NotFound desc = could not find container \"d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8\": container with ID starting with d590d3d86d4c92d6c7ca7970571107c745de1c90b9dcfe9367f5e684113345c8 not found: ID does not exist" Sep 11 14:19:49.700533 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.700438 2579 scope.go:117] "RemoveContainer" containerID="9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37" Sep 11 14:19:49.700882 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:19:49.700862 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37\": container with ID starting with 9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37 not found: ID does not exist" containerID="9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37" Sep 11 14:19:49.700951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.700892 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37"} err="failed to get container status \"9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37\": rpc error: code = NotFound desc = could not find container \"9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37\": container with ID starting with 9a667e9a9388b7439ad02f652a5020b3e2982f51c306042c43c0ace7df473e37 not found: ID does not exist" Sep 11 14:19:49.704671 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.704650 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4"] Sep 11 14:19:49.718819 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.718797 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:49.720510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.720488 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-106468-openshift-default-7c4f4f8bbd-d9xs4"] Sep 11 14:19:49.737128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.737076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fe1636-5684-40a1-8a15-54685f37821c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.737218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.737169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4js8w\" (UniqueName: \"kubernetes.io/projected/68fe1636-5684-40a1-8a15-54685f37821c-kube-api-access-4js8w\") pod \"payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.737731 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.737714 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fe1636-5684-40a1-8a15-54685f37821c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.749359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.749339 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4js8w\" (UniqueName: \"kubernetes.io/projected/68fe1636-5684-40a1-8a15-54685f37821c-kube-api-access-4js8w\") pod \"payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.804711 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.804680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:49.954135 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:49.954028 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:50.095712 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:50.095667 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk"] Sep 11 14:19:50.097060 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:19:50.097025 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fe1636_5684_40a1_8a15_54685f37821c.slice/crio-fecda9114742f79bf47e212ea39b8ef336a4e7fdf02bc6f06825b0073f641ae2 WatchSource:0}: Error finding container fecda9114742f79bf47e212ea39b8ef336a4e7fdf02bc6f06825b0073f641ae2: Status 404 returned error can't find the container with id fecda9114742f79bf47e212ea39b8ef336a4e7fdf02bc6f06825b0073f641ae2 Sep 11 14:19:50.665041 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:50.665004 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" event={"ID":"68fe1636-5684-40a1-8a15-54685f37821c","Type":"ContainerStarted","Data":"62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1"} Sep 11 14:19:50.665508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:50.665051 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" event={"ID":"68fe1636-5684-40a1-8a15-54685f37821c","Type":"ContainerStarted","Data":"fecda9114742f79bf47e212ea39b8ef336a4e7fdf02bc6f06825b0073f641ae2"} Sep 11 14:19:50.665508 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:50.665229 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:19:50.688840 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:50.688787 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" podStartSLOduration=1.6887672 podStartE2EDuration="1.6887672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:19:50.687317823 +0000 UTC m=+1607.557049085" watchObservedRunningTime="2026-09-11 14:19:50.6887672 +0000 UTC m=+1607.558498462" Sep 11 14:19:51.162335 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:51.162297 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:51.644344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:51.644311 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" path="/var/lib/kubelet/pods/0ca7778f-a232-4ff6-ab16-423c9dd7fcef/volumes" Sep 11 14:19:52.365084 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:52.365044 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:53.463445 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:53.461446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:54.530245 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.530207 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:54.585128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.584719 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x"] Sep 11 14:19:54.601227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.601192 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-c52865-29818935-bxb8x"] Sep 11 14:19:54.681734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.681693 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp"] Sep 11 14:19:54.686127 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.682236 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" podUID="853e1c77-b259-42b4-a48a-210d5037882d" containerName="payload-pre-processing" containerID="cri-o://6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4" gracePeriod=30 Sep 11 14:19:54.754569 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.754525 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:54.893787 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:54.893696 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:55.001426 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.001391 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx"] Sep 11 14:19:55.009498 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.009467 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c52865-v4mnx"] Sep 11 14:19:55.015337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.015313 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:55.029825 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.029801 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:19:55.129349 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.129259 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:55.205743 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.205707 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/853e1c77-b259-42b4-a48a-210d5037882d-kube-api-access-r89vf\") pod \"853e1c77-b259-42b4-a48a-210d5037882d\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " Sep 11 14:19:55.205946 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.205816 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853e1c77-b259-42b4-a48a-210d5037882d-plugins-config-volume\") pod \"853e1c77-b259-42b4-a48a-210d5037882d\" (UID: \"853e1c77-b259-42b4-a48a-210d5037882d\") " Sep 11 14:19:55.206186 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.206163 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/853e1c77-b259-42b4-a48a-210d5037882d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "853e1c77-b259-42b4-a48a-210d5037882d" (UID: "853e1c77-b259-42b4-a48a-210d5037882d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:19:55.207891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.207872 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/853e1c77-b259-42b4-a48a-210d5037882d-kube-api-access-r89vf" (OuterVolumeSpecName: "kube-api-access-r89vf") pod "853e1c77-b259-42b4-a48a-210d5037882d" (UID: "853e1c77-b259-42b4-a48a-210d5037882d"). InnerVolumeSpecName "kube-api-access-r89vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:19:55.306690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.306647 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/853e1c77-b259-42b4-a48a-210d5037882d-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:55.306690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.306680 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r89vf\" (UniqueName: \"kubernetes.io/projected/853e1c77-b259-42b4-a48a-210d5037882d-kube-api-access-r89vf\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:19:55.361497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.361453 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:55.574146 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.574084 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:55.644233 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.644192 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90f3bdc3-0e8e-481d-bbe1-24c4362a0896" path="/var/lib/kubelet/pods/90f3bdc3-0e8e-481d-bbe1-24c4362a0896/volumes" Sep 11 14:19:55.644512 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.644499 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd3db8b-37ee-4a7a-9d77-215d0ff669ce" path="/var/lib/kubelet/pods/ebd3db8b-37ee-4a7a-9d77-215d0ff669ce/volumes" Sep 11 14:19:55.712433 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.712401 2579 generic.go:356] "Generic (PLEG): container finished" podID="853e1c77-b259-42b4-a48a-210d5037882d" containerID="6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4" exitCode=0 Sep 11 14:19:55.712613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.712447 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" Sep 11 14:19:55.712613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.712484 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" event={"ID":"853e1c77-b259-42b4-a48a-210d5037882d","Type":"ContainerDied","Data":"6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4"} Sep 11 14:19:55.712613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.712548 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp" event={"ID":"853e1c77-b259-42b4-a48a-210d5037882d","Type":"ContainerDied","Data":"0f3949df6ab28430704feed4e35efe136ff81700f3d47ff66d69345c24e7d31d"} Sep 11 14:19:55.712613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.712577 2579 scope.go:117] "RemoveContainer" containerID="6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4" Sep 11 14:19:55.722584 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.722564 2579 scope.go:117] "RemoveContainer" containerID="6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4" Sep 11 14:19:55.722928 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:19:55.722907 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4\": container with ID starting with 6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4 not found: ID does not exist" containerID="6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4" Sep 11 14:19:55.722994 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.722936 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4"} err="failed to get container status \"6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4\": rpc error: code = NotFound desc = could not find container \"6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4\": container with ID starting with 6a69bf7edc04d740bbe55c9801e3eeba15dead9086bf9200826fdf330b8241a4 not found: ID does not exist" Sep 11 14:19:55.748436 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.748408 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp"] Sep 11 14:19:55.763807 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:55.763781 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-c52865-dbb4db686-l5rwp"] Sep 11 14:19:56.655233 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:56.655195 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:57.643965 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:57.643936 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="853e1c77-b259-42b4-a48a-210d5037882d" path="/var/lib/kubelet/pods/853e1c77-b259-42b4-a48a-210d5037882d/volumes" Sep 11 14:19:57.894170 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:57.894023 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:19:59.287703 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:19:59.287671 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:00.235722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:00.235688 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:01.673428 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:01.673401 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:20:02.666153 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:02.666119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:02.754450 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:02.754417 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:03.152481 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.152446 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:03.488625 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.488540 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:03.592608 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.592567 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:03.772539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.772454 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerID="33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68" exitCode=0 Sep 11 14:20:03.772539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.772528 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" event={"ID":"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1","Type":"ContainerDied","Data":"33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68"} Sep 11 14:20:03.772986 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.772972 2579 scope.go:117] "RemoveContainer" containerID="33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68" Sep 11 14:20:03.774278 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.774256 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:03.774337 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.774324 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:03.774373 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.774354 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:20:03.789038 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.789016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:03.946038 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:03.946000 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:04.050457 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:04.050426 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:04.756313 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:04.756276 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:04.788464 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:04.788407 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" event={"ID":"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1","Type":"ContainerStarted","Data":"d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17"} Sep 11 14:20:05.197301 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:05.197263 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:20:05.197575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:05.197553 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:05.197625 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:05.197603 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 11 14:20:05.942379 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:05.942351 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:06.198280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:06.198188 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:06.198280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:06.198256 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 11 14:20:07.197819 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:07.197779 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:07.198336 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:07.197843 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 11 14:20:07.202826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:07.202789 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:07.363319 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:07.363260 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:07.510892 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:07.510774 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:08.197634 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:08.197595 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:08.197817 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:08.197689 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 11 14:20:08.731587 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:08.731548 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:09.198516 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:09.198479 2579 patch_prober.go:28] interesting pod/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Sep 11 14:20:09.198712 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:09.198585 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Sep 11 14:20:09.866552 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:09.866512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:10.201696 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:10.201600 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:20:10.202069 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:10.202047 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:20:10.202807 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:10.202790 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:20:11.041627 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:11.041536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:11.148495 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:11.148465 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:11.255489 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:11.255451 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:11.693976 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:11.693928 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:11.841452 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:11.841408 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:12.423071 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:12.423037 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:13.435866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:13.435830 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:14.235787 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:14.235750 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:14.331666 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:14.331631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:15.625304 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:15.625270 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:16.386053 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.386019 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:16.448409 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.448369 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n"] Sep 11 14:20:16.448675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.448646 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" podUID="c2d775cc-2682-418e-a873-9ea29769f6f8" containerName="payload-processing" containerID="cri-o://65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922" gracePeriod=30 Sep 11 14:20:16.502043 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.502007 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:16.607783 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.607743 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:16.721590 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.721489 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:16.761401 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.761372 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:20:16.852870 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.852836 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d775cc-2682-418e-a873-9ea29769f6f8-plugins-config-volume\") pod \"c2d775cc-2682-418e-a873-9ea29769f6f8\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " Sep 11 14:20:16.853040 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.852906 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjwtg\" (UniqueName: \"kubernetes.io/projected/c2d775cc-2682-418e-a873-9ea29769f6f8-kube-api-access-mjwtg\") pod \"c2d775cc-2682-418e-a873-9ea29769f6f8\" (UID: \"c2d775cc-2682-418e-a873-9ea29769f6f8\") " Sep 11 14:20:16.853316 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.853292 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d775cc-2682-418e-a873-9ea29769f6f8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2d775cc-2682-418e-a873-9ea29769f6f8" (UID: "c2d775cc-2682-418e-a873-9ea29769f6f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:20:16.855262 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.855240 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d775cc-2682-418e-a873-9ea29769f6f8-kube-api-access-mjwtg" (OuterVolumeSpecName: "kube-api-access-mjwtg") pod "c2d775cc-2682-418e-a873-9ea29769f6f8" (UID: "c2d775cc-2682-418e-a873-9ea29769f6f8"). InnerVolumeSpecName "kube-api-access-mjwtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:16.877839 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.877812 2579 generic.go:356] "Generic (PLEG): container finished" podID="c2d775cc-2682-418e-a873-9ea29769f6f8" containerID="65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922" exitCode=0 Sep 11 14:20:16.877957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.877854 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" Sep 11 14:20:16.877957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.877888 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" event={"ID":"c2d775cc-2682-418e-a873-9ea29769f6f8","Type":"ContainerDied","Data":"65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922"} Sep 11 14:20:16.877957 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.877938 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n" event={"ID":"c2d775cc-2682-418e-a873-9ea29769f6f8","Type":"ContainerDied","Data":"5252c594ce640d4ac4aa6a63a4db6bb67c4cc383998194cb1875f3590f63d144"} Sep 11 14:20:16.878064 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.877961 2579 scope.go:117] "RemoveContainer" containerID="65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922" Sep 11 14:20:16.888147 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.888129 2579 scope.go:117] "RemoveContainer" containerID="65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922" Sep 11 14:20:16.888437 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:20:16.888414 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922\": container with ID starting with 65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922 not found: ID does not exist" containerID="65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922" Sep 11 14:20:16.888491 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.888447 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922"} err="failed to get container status \"65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922\": rpc error: code = NotFound desc = could not find container \"65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922\": container with ID starting with 65238d9da0bd0a34ddd94dcf4a96f5f43f1639e2cd05abc56b8d1b86652da922 not found: ID does not exist" Sep 11 14:20:16.904046 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.904013 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n"] Sep 11 14:20:16.910335 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.910313 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5cf565-69fd7d4787-52t7n"] Sep 11 14:20:16.954538 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.954503 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2d775cc-2682-418e-a873-9ea29769f6f8-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:16.954538 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:16.954532 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjwtg\" (UniqueName: \"kubernetes.io/projected/c2d775cc-2682-418e-a873-9ea29769f6f8-kube-api-access-mjwtg\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:17.084500 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:17.084469 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:17.337214 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:17.337097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:17.644078 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:17.644000 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d775cc-2682-418e-a873-9ea29769f6f8" path="/var/lib/kubelet/pods/c2d775cc-2682-418e-a873-9ea29769f6f8/volumes" Sep 11 14:20:18.304000 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:18.303966 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:19.351114 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:19.351080 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:20.299018 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:20.298975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:21.341670 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:21.341631 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:22.145161 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:22.145095 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:22.612896 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:22.612856 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:23.741282 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:23.741251 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:24.748763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:24.748723 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:25.209553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:25.209514 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:25.327789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:25.327752 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:25.416178 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:25.416143 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx"] Sep 11 14:20:25.416410 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:25.416384 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" containerID="cri-o://3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a" gracePeriod=30 Sep 11 14:20:25.462982 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:25.462887 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:25.769631 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:25.769536 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:26.581575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:26.581531 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:30.672795 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.672766 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:20:30.693971 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.693902 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6skw4\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-kube-api-access-6skw4\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.693971 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.693941 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-token\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694211 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.693996 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-socket\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694211 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694046 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-certs\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694211 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694070 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-data\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694211 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694122 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-podinfo\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694211 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694164 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-envoy\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694211 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694198 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-credential-socket\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694533 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istiod-ca-cert\") pod \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\" (UID: \"d865c57c-3c7a-4d7c-b1fa-78163296f3ad\") " Sep 11 14:20:30.694533 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694388 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:30.694533 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694399 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:30.694533 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694411 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-data" (OuterVolumeSpecName: "istio-data") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:30.694735 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694587 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:30.694735 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.694716 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:20:30.695078 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.695058 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.695595 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.695569 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.695707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.695602 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.695707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.695617 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.695707 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.695631 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.696803 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.696782 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:20:30.697043 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.697027 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-kube-api-access-6skw4" (OuterVolumeSpecName: "kube-api-access-6skw4") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "kube-api-access-6skw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:30.697272 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.697245 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-token" (OuterVolumeSpecName: "istio-token") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:20:30.697596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.697573 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d865c57c-3c7a-4d7c-b1fa-78163296f3ad" (UID: "d865c57c-3c7a-4d7c-b1fa-78163296f3ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:20:30.796239 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.796204 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6skw4\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-kube-api-access-6skw4\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.796239 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.796232 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.796239 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.796244 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.796239 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.796252 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d865c57c-3c7a-4d7c-b1fa-78163296f3ad-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:20:30.972010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.971914 2579 generic.go:356] "Generic (PLEG): container finished" podID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerID="3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a" exitCode=0 Sep 11 14:20:30.972010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.971961 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" Sep 11 14:20:30.972010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.971999 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" event={"ID":"d865c57c-3c7a-4d7c-b1fa-78163296f3ad","Type":"ContainerDied","Data":"3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a"} Sep 11 14:20:30.972320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.972055 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx" event={"ID":"d865c57c-3c7a-4d7c-b1fa-78163296f3ad","Type":"ContainerDied","Data":"4fd44b7ed44b5032b87fd70d02421502d31272c5c436c710ee15a6f0406372e7"} Sep 11 14:20:30.972320 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.972077 2579 scope.go:117] "RemoveContainer" containerID="3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a" Sep 11 14:20:30.982670 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.982651 2579 scope.go:117] "RemoveContainer" containerID="a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985" Sep 11 14:20:30.991967 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.991951 2579 scope.go:117] "RemoveContainer" containerID="3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a" Sep 11 14:20:30.992258 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:20:30.992242 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a\": container with ID starting with 3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a not found: ID does not exist" containerID="3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a" Sep 11 14:20:30.992312 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.992265 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a"} err="failed to get container status \"3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a\": rpc error: code = NotFound desc = could not find container \"3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a\": container with ID starting with 3eeac1fbf1cc000c86c75350a19eb489c87e794948c3effb085c7f4e40966b6a not found: ID does not exist" Sep 11 14:20:30.992312 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.992281 2579 scope.go:117] "RemoveContainer" containerID="a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985" Sep 11 14:20:30.992520 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:20:30.992499 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985\": container with ID starting with a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985 not found: ID does not exist" containerID="a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985" Sep 11 14:20:30.992566 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:30.992523 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985"} err="failed to get container status \"a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985\": rpc error: code = NotFound desc = could not find container \"a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985\": container with ID starting with a213a5f715c4575d4891cece139666c481f2588b327ffa15c0862dece4c0d985 not found: ID does not exist" Sep 11 14:20:31.000706 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:31.000678 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx"] Sep 11 14:20:31.003384 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:31.003364 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5cf565-openshift-default-67bf8c5bf6-59rdx"] Sep 11 14:20:31.644344 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:31.644313 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" path="/var/lib/kubelet/pods/d865c57c-3c7a-4d7c-b1fa-78163296f3ad/volumes" Sep 11 14:20:32.241265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:32.241226 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:20:42.148520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:20:42.148479 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:04.026636 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:04.026595 2579 scope.go:117] "RemoveContainer" containerID="c77dc75059452e461180974b15ba77bfc8c518cf6c1624779a4f570fb10ec009" Sep 11 14:21:04.035813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:04.035792 2579 scope.go:117] "RemoveContainer" containerID="648172f4c6ef4f205d297f40a054bf604e30ea4821d155048545e2795e419bb2" Sep 11 14:21:08.126250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:08.126215 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk_68fe1636-5684-40a1-8a15-54685f37821c/payload-processing/0.log" Sep 11 14:21:08.720556 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:08.720516 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:09.035398 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035322 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw"] Sep 11 14:21:09.035796 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035781 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="853e1c77-b259-42b4-a48a-210d5037882d" containerName="payload-pre-processing" Sep 11 14:21:09.035847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035799 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="853e1c77-b259-42b4-a48a-210d5037882d" containerName="payload-pre-processing" Sep 11 14:21:09.035847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035812 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" Sep 11 14:21:09.035847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035819 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" Sep 11 14:21:09.035847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035835 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d775cc-2682-418e-a873-9ea29769f6f8" containerName="payload-processing" Sep 11 14:21:09.035847 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035841 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d775cc-2682-418e-a873-9ea29769f6f8" containerName="payload-processing" Sep 11 14:21:09.036027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035914 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" Sep 11 14:21:09.036027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035921 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d775cc-2682-418e-a873-9ea29769f6f8" containerName="payload-processing" Sep 11 14:21:09.036027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035926 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" Sep 11 14:21:09.036027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035933 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="853e1c77-b259-42b4-a48a-210d5037882d" containerName="payload-pre-processing" Sep 11 14:21:09.036027 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.035941 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ca7778f-a232-4ff6-ab16-423c9dd7fcef" containerName="istio-proxy" Sep 11 14:21:09.040552 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.040532 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.043665 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.043643 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-246a87-openshift-default-dockercfg-6rx6t\"" Sep 11 14:21:09.057613 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.057587 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-socket\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.057749 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.057718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-credential-socket\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.057813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.057754 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vjl8\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-kube-api-access-5vjl8\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.057813 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.057786 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-podinfo\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.057929 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.057884 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-certs\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.057990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.057964 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08225ba2-19d7-4234-91c6-3d2089aaa971-istiod-ca-cert\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.058051 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.058007 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-envoy\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.058051 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.058042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-data\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.058187 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.058068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-token\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.063790 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.063765 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw"] Sep 11 14:21:09.120848 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.120812 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:09.159721 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.159687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-data\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.159736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-token\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.159775 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-socket\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.159881 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-credential-socket\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.159920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vjl8\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-kube-api-access-5vjl8\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.159964 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-podinfo\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-certs\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08225ba2-19d7-4234-91c6-3d2089aaa971-istiod-ca-cert\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-envoy\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160166 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-data\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160809 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160294 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-credential-socket\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160809 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160361 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-socket\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160809 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160534 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-certs\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.160966 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.160865 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08225ba2-19d7-4234-91c6-3d2089aaa971-istiod-ca-cert\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.162505 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.162473 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-envoy\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.162646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.162628 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-podinfo\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.178917 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.178898 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-token\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.182844 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.182827 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vjl8\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-kube-api-access-5vjl8\") pod \"e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.351206 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.351169 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:09.611077 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.610984 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:09.646393 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:21:09.646348 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08225ba2_19d7_4234_91c6_3d2089aaa971.slice/crio-d92a6de852d3d42f31f1285630a3287be13eddca3f0bc93acaccb755b69607ec WatchSource:0}: Error finding container d92a6de852d3d42f31f1285630a3287be13eddca3f0bc93acaccb755b69607ec: Status 404 returned error can't find the container with id d92a6de852d3d42f31f1285630a3287be13eddca3f0bc93acaccb755b69607ec Sep 11 14:21:09.648815 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.648782 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:21:09.648927 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.648857 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:21:09.648927 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.648902 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:21:09.651177 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:09.651158 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw"] Sep 11 14:21:10.222332 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:10.222294 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" event={"ID":"08225ba2-19d7-4234-91c6-3d2089aaa971","Type":"ContainerStarted","Data":"d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab"} Sep 11 14:21:10.222715 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:10.222357 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" event={"ID":"08225ba2-19d7-4234-91c6-3d2089aaa971","Type":"ContainerStarted","Data":"d92a6de852d3d42f31f1285630a3287be13eddca3f0bc93acaccb755b69607ec"} Sep 11 14:21:10.352224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:10.352174 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:10.357437 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:10.357408 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:10.416252 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:10.416203 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podStartSLOduration=1.416186312 podStartE2EDuration="1.416186312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:21:10.260927098 +0000 UTC m=+1687.130658358" watchObservedRunningTime="2026-09-11 14:21:10.416186312 +0000 UTC m=+1687.285917584" Sep 11 14:21:11.229257 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:11.229224 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:11.230560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:11.230539 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:11.640561 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:11.640526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:12.229762 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:12.229727 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:12.319636 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:12.319607 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:12.600703 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:12.600669 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:14.007639 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:14.007603 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:15.041414 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:15.041378 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:16.007064 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:16.006947 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:16.974974 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:16.974936 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:17.097417 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:17.097379 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:17.228172 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:17.228048 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:17.772412 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:17.772382 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:17.881684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:17.881656 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:32.367419 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:32.367385 2579 generic.go:356] "Generic (PLEG): container finished" podID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerID="d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab" exitCode=0 Sep 11 14:21:32.367880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:32.367469 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" event={"ID":"08225ba2-19d7-4234-91c6-3d2089aaa971","Type":"ContainerDied","Data":"d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab"} Sep 11 14:21:32.367930 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:32.367880 2579 scope.go:117] "RemoveContainer" containerID="d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab" Sep 11 14:21:32.369077 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:32.369049 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:21:32.369207 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:32.369171 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:21:32.369277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:32.369223 2579 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892148Ki","pods":"250"} Sep 11 14:21:33.389829 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:33.389794 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" event={"ID":"08225ba2-19d7-4234-91c6-3d2089aaa971","Type":"ContainerStarted","Data":"0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1"} Sep 11 14:21:34.351861 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:34.351826 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:34.352148 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:34.352040 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:21:34.352223 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:34.352136 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 11 14:21:35.351869 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:35.351833 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:21:35.352277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:35.351919 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 11 14:21:36.352422 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:36.352390 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:21:36.352810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:36.352444 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 11 14:21:37.351603 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:37.351567 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:21:37.351806 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:37.351625 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 11 14:21:38.351541 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:38.351504 2579 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Sep 11 14:21:38.352000 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:38.351574 2579 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Sep 11 14:21:39.356587 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:39.356556 2579 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:39.356962 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:39.356940 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:39.357716 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:39.357697 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:21:39.745677 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:39.745584 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:40.728202 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:40.728157 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:41.801498 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:41.801460 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:42.785459 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:42.785433 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:42.913479 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:42.913443 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:43.553415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.553384 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh"] Sep 11 14:21:43.553822 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.553810 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" Sep 11 14:21:43.553865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.553825 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d865c57c-3c7a-4d7c-b1fa-78163296f3ad" containerName="istio-proxy" Sep 11 14:21:43.557171 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.557156 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:43.562795 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.562767 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:21:43.563326 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.563310 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:21:43.580277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.580242 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh"] Sep 11 14:21:43.700237 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.700204 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea2ea4-60e2-455f-a330-38cde4feffcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh\" (UID: \"d7ea2ea4-60e2-455f-a330-38cde4feffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:43.801173 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.801140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea2ea4-60e2-455f-a330-38cde4feffcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh\" (UID: \"d7ea2ea4-60e2-455f-a330-38cde4feffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:43.801762 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.801741 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea2ea4-60e2-455f-a330-38cde4feffcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh\" (UID: \"d7ea2ea4-60e2-455f-a330-38cde4feffcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:43.868515 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.868425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:43.882834 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:43.882807 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:44.149565 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:44.149535 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh"] Sep 11 14:21:44.149926 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:21:44.149867 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7ea2ea4_60e2_455f_a330_38cde4feffcc.slice/crio-133c3e9a0f936104c88da8047cf757c4fa6b3e6ea99f1aa6a6ff626edaa64748 WatchSource:0}: Error finding container 133c3e9a0f936104c88da8047cf757c4fa6b3e6ea99f1aa6a6ff626edaa64748: Status 404 returned error can't find the container with id 133c3e9a0f936104c88da8047cf757c4fa6b3e6ea99f1aa6a6ff626edaa64748 Sep 11 14:21:44.462510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:44.462432 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" event={"ID":"d7ea2ea4-60e2-455f-a330-38cde4feffcc","Type":"ContainerStarted","Data":"133c3e9a0f936104c88da8047cf757c4fa6b3e6ea99f1aa6a6ff626edaa64748"} Sep 11 14:21:44.836510 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:44.836444 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:45.470176 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:45.470146 2579 generic.go:356] "Generic (PLEG): container finished" podID="d7ea2ea4-60e2-455f-a330-38cde4feffcc" containerID="ee4720eb6abc4905f631df5143212d878ddb04de18ec1134be1da1543e971931" exitCode=0 Sep 11 14:21:45.470550 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:45.470195 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" event={"ID":"d7ea2ea4-60e2-455f-a330-38cde4feffcc","Type":"ContainerDied","Data":"ee4720eb6abc4905f631df5143212d878ddb04de18ec1134be1da1543e971931"} Sep 11 14:21:45.727786 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:45.727681 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:46.562828 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:46.562782 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:46.628534 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:46.628506 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:46.729254 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:46.729177 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea2ea4-60e2-455f-a330-38cde4feffcc-maas-api-service-ca\") pod \"d7ea2ea4-60e2-455f-a330-38cde4feffcc\" (UID: \"d7ea2ea4-60e2-455f-a330-38cde4feffcc\") " Sep 11 14:21:46.729578 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:46.729552 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7ea2ea4-60e2-455f-a330-38cde4feffcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7ea2ea4-60e2-455f-a330-38cde4feffcc" (UID: "d7ea2ea4-60e2-455f-a330-38cde4feffcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:21:46.830442 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:46.830401 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7ea2ea4-60e2-455f-a330-38cde4feffcc-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:21:47.483930 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:47.483899 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" Sep 11 14:21:47.483930 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:47.483927 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh" event={"ID":"d7ea2ea4-60e2-455f-a330-38cde4feffcc","Type":"ContainerDied","Data":"133c3e9a0f936104c88da8047cf757c4fa6b3e6ea99f1aa6a6ff626edaa64748"} Sep 11 14:21:47.484237 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:47.483968 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="133c3e9a0f936104c88da8047cf757c4fa6b3e6ea99f1aa6a6ff626edaa64748" Sep 11 14:21:47.547477 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:47.547442 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:48.355193 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:48.355151 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:49.231023 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:49.230975 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:53.780948 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:53.780909 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.490625 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.490586 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.618720 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.618677 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.757928 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.757835 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.860130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.860084 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh"] Sep 11 14:21:54.863629 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.863603 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-246a87-l6gdh"] Sep 11 14:21:54.865507 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.865484 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:54.946789 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:54.946745 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:55.253471 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:55.253434 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:55.600507 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:55.600464 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:55.643192 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:55.643157 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7ea2ea4-60e2-455f-a330-38cde4feffcc" path="/var/lib/kubelet/pods/d7ea2ea4-60e2-455f-a330-38cde4feffcc/volumes" Sep 11 14:21:56.398811 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:56.398768 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:56.490130 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:56.490072 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:56.617550 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:56.617503 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:57.676436 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:57.676381 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:58.542530 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:58.542494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:21:59.432537 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:21:59.432494 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.029887 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:02.029842 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.138220 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:02.138190 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:02.283681 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:02.283589 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw"] Sep 11 14:22:02.283866 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:02.283812 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" containerID="cri-o://0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1" gracePeriod=30 Sep 11 14:22:02.321330 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:02.321296 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:05.988125 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.988072 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8"] Sep 11 14:22:05.988586 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.988571 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7ea2ea4-60e2-455f-a330-38cde4feffcc" containerName="revoke-keys" Sep 11 14:22:05.988650 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.988587 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7ea2ea4-60e2-455f-a330-38cde4feffcc" containerName="revoke-keys" Sep 11 14:22:05.988691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.988656 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7ea2ea4-60e2-455f-a330-38cde4feffcc" containerName="revoke-keys" Sep 11 14:22:05.991765 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.991750 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:05.996607 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.996583 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:22:05.996832 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:05.996818 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:22:06.008185 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.008163 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8"] Sep 11 14:22:06.135001 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.134960 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8\" (UID: \"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:06.236242 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.236209 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8\" (UID: \"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:06.236835 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.236818 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8\" (UID: \"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:06.248560 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.248478 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:06.301531 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.301503 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:06.588638 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.588615 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8"] Sep 11 14:22:06.590498 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:22:06.590470 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd2816b_b1d2_4b8b_adc3_3aff5272a6a8.slice/crio-a2df07ca31dd23308585c0c2f293913b0a878368ce8157c655e4512f90bb4596 WatchSource:0}: Error finding container a2df07ca31dd23308585c0c2f293913b0a878368ce8157c655e4512f90bb4596: Status 404 returned error can't find the container with id a2df07ca31dd23308585c0c2f293913b0a878368ce8157c655e4512f90bb4596 Sep 11 14:22:06.613279 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:06.613248 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" event={"ID":"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8","Type":"ContainerStarted","Data":"a2df07ca31dd23308585c0c2f293913b0a878368ce8157c655e4512f90bb4596"} Sep 11 14:22:07.262926 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.262834 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:07.537431 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.537405 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:22:07.621755 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.621714 2579 generic.go:356] "Generic (PLEG): container finished" podID="ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8" containerID="290406f5a23fd6af96c155a13e8be71576fea733778202e7bfdc085e01078bb2" exitCode=0 Sep 11 14:22:07.621953 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.621799 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" event={"ID":"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8","Type":"ContainerDied","Data":"290406f5a23fd6af96c155a13e8be71576fea733778202e7bfdc085e01078bb2"} Sep 11 14:22:07.623734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.623709 2579 generic.go:356] "Generic (PLEG): container finished" podID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerID="0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1" exitCode=0 Sep 11 14:22:07.623870 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.623760 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" Sep 11 14:22:07.623870 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.623793 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" event={"ID":"08225ba2-19d7-4234-91c6-3d2089aaa971","Type":"ContainerDied","Data":"0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1"} Sep 11 14:22:07.623870 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.623843 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw" event={"ID":"08225ba2-19d7-4234-91c6-3d2089aaa971","Type":"ContainerDied","Data":"d92a6de852d3d42f31f1285630a3287be13eddca3f0bc93acaccb755b69607ec"} Sep 11 14:22:07.623870 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.623869 2579 scope.go:117] "RemoveContainer" containerID="0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1" Sep 11 14:22:07.634404 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.634376 2579 scope.go:117] "RemoveContainer" containerID="d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab" Sep 11 14:22:07.644568 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.644534 2579 scope.go:117] "RemoveContainer" containerID="0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1" Sep 11 14:22:07.644882 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:22:07.644852 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1\": container with ID starting with 0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1 not found: ID does not exist" containerID="0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1" Sep 11 14:22:07.644984 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.644893 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1"} err="failed to get container status \"0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1\": rpc error: code = NotFound desc = could not find container \"0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1\": container with ID starting with 0ae50a3ba1da1d202c5701132917ebe5a894bcabc671d2accce39c0bc7d36fa1 not found: ID does not exist" Sep 11 14:22:07.644984 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.644916 2579 scope.go:117] "RemoveContainer" containerID="d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab" Sep 11 14:22:07.645238 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:22:07.645215 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab\": container with ID starting with d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab not found: ID does not exist" containerID="d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab" Sep 11 14:22:07.645295 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.645246 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab"} err="failed to get container status \"d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab\": rpc error: code = NotFound desc = could not find container \"d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab\": container with ID starting with d9543a4b302444daac860fe79b12cdea588b3295792479ff8f1dd3ada9b2d1ab not found: ID does not exist" Sep 11 14:22:07.651868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.651845 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-socket\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.651990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.651916 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-credential-socket\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.651990 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.651943 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-data\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652189 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.651990 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-certs\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652189 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652048 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-token\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652189 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652142 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:22:07.652415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652191 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vjl8\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-kube-api-access-5vjl8\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652245 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-podinfo\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-envoy\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652343 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08225ba2-19d7-4234-91c6-3d2089aaa971-istiod-ca-cert\") pod \"08225ba2-19d7-4234-91c6-3d2089aaa971\" (UID: \"08225ba2-19d7-4234-91c6-3d2089aaa971\") " Sep 11 14:22:07.652690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652273 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:22:07.652690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652462 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-data" (OuterVolumeSpecName: "istio-data") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:22:07.652690 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652482 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:22:07.652850 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652757 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08225ba2-19d7-4234-91c6-3d2089aaa971-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:22:07.652907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652852 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.652907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652873 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.652907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652887 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.652907 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.652900 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.654577 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.654549 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-token" (OuterVolumeSpecName: "istio-token") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:22:07.654755 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.654731 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-kube-api-access-5vjl8" (OuterVolumeSpecName: "kube-api-access-5vjl8") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "kube-api-access-5vjl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:22:07.655076 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.655046 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:22:07.655183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.655147 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08225ba2-19d7-4234-91c6-3d2089aaa971" (UID: "08225ba2-19d7-4234-91c6-3d2089aaa971"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:22:07.753520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.753488 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vjl8\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-kube-api-access-5vjl8\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.753520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.753520 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.753520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.753530 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.753749 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.753538 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08225ba2-19d7-4234-91c6-3d2089aaa971-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.753749 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.753546 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08225ba2-19d7-4234-91c6-3d2089aaa971-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:07.951814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.951774 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw"] Sep 11 14:22:07.959378 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:07.959348 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-246a87-openshift-default-676cdcdbbd-g9jtw"] Sep 11 14:22:08.346939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:08.346907 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:08.771114 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:08.771084 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:08.864165 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:08.864136 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8-maas-api-service-ca\") pod \"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8\" (UID: \"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8\") " Sep 11 14:22:08.864519 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:08.864480 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8" (UID: "ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:22:08.965311 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:08.965231 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:09.154597 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:09.154549 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:09.639578 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:09.639548 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" Sep 11 14:22:09.643264 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:09.643236 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" path="/var/lib/kubelet/pods/08225ba2-19d7-4234-91c6-3d2089aaa971/volumes" Sep 11 14:22:09.643747 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:09.643723 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8" event={"ID":"ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8","Type":"ContainerDied","Data":"a2df07ca31dd23308585c0c2f293913b0a878368ce8157c655e4512f90bb4596"} Sep 11 14:22:09.643890 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:09.643758 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2df07ca31dd23308585c0c2f293913b0a878368ce8157c655e4512f90bb4596" Sep 11 14:22:10.184978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:10.184942 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:11.260787 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:11.260749 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:12.203152 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:12.203092 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:16.150512 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:16.150468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:21.932766 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:21.932721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:21.976094 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:21.976058 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk"] Sep 11 14:22:21.976391 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:21.976355 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" podUID="68fe1636-5684-40a1-8a15-54685f37821c" containerName="payload-processing" containerID="cri-o://62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1" gracePeriod=30 Sep 11 14:22:22.032479 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.032447 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:22.133989 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.133938 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:22.217817 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.217649 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8"] Sep 11 14:22:22.218891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.218861 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e6521d-nh5t8"] Sep 11 14:22:22.234780 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.234737 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:22.284396 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.284371 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:22:22.302801 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.302775 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4js8w\" (UniqueName: \"kubernetes.io/projected/68fe1636-5684-40a1-8a15-54685f37821c-kube-api-access-4js8w\") pod \"68fe1636-5684-40a1-8a15-54685f37821c\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " Sep 11 14:22:22.302959 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.302884 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fe1636-5684-40a1-8a15-54685f37821c-plugins-config-volume\") pod \"68fe1636-5684-40a1-8a15-54685f37821c\" (UID: \"68fe1636-5684-40a1-8a15-54685f37821c\") " Sep 11 14:22:22.303287 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.303259 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68fe1636-5684-40a1-8a15-54685f37821c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "68fe1636-5684-40a1-8a15-54685f37821c" (UID: "68fe1636-5684-40a1-8a15-54685f37821c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:22:22.305010 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.304987 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fe1636-5684-40a1-8a15-54685f37821c-kube-api-access-4js8w" (OuterVolumeSpecName: "kube-api-access-4js8w") pod "68fe1636-5684-40a1-8a15-54685f37821c" (UID: "68fe1636-5684-40a1-8a15-54685f37821c"). InnerVolumeSpecName "kube-api-access-4js8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:22:22.404154 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.404076 2579 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68fe1636-5684-40a1-8a15-54685f37821c-plugins-config-volume\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:22.404154 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.404145 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4js8w\" (UniqueName: \"kubernetes.io/projected/68fe1636-5684-40a1-8a15-54685f37821c-kube-api-access-4js8w\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:22:22.627604 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.627565 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:22.722660 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.722619 2579 generic.go:356] "Generic (PLEG): container finished" podID="68fe1636-5684-40a1-8a15-54685f37821c" containerID="62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1" exitCode=0 Sep 11 14:22:22.722842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.722678 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" Sep 11 14:22:22.722842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.722700 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" event={"ID":"68fe1636-5684-40a1-8a15-54685f37821c","Type":"ContainerDied","Data":"62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1"} Sep 11 14:22:22.722842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.722756 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk" event={"ID":"68fe1636-5684-40a1-8a15-54685f37821c","Type":"ContainerDied","Data":"fecda9114742f79bf47e212ea39b8ef336a4e7fdf02bc6f06825b0073f641ae2"} Sep 11 14:22:22.722842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.722779 2579 scope.go:117] "RemoveContainer" containerID="62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1" Sep 11 14:22:22.733555 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.733526 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:22.734133 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.734081 2579 scope.go:117] "RemoveContainer" containerID="62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1" Sep 11 14:22:22.734401 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:22:22.734379 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1\": container with ID starting with 62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1 not found: ID does not exist" containerID="62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1" Sep 11 14:22:22.734461 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.734413 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1"} err="failed to get container status \"62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1\": rpc error: code = NotFound desc = could not find container \"62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1\": container with ID starting with 62ff2860712511a64cd92d6973e6110905db1489e5e673894fabb92c32a6fde1 not found: ID does not exist" Sep 11 14:22:22.752491 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.752463 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk"] Sep 11 14:22:22.757151 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:22.757122 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e6521d-74b7f57c79-zf7bk"] Sep 11 14:22:23.644675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:23.644639 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fe1636-5684-40a1-8a15-54685f37821c" path="/var/lib/kubelet/pods/68fe1636-5684-40a1-8a15-54685f37821c/volumes" Sep 11 14:22:23.645221 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:23.645198 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8" path="/var/lib/kubelet/pods/ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8/volumes" Sep 11 14:22:23.646785 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:23.646761 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:24.414400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:24.414353 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:25.374474 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:25.374431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:27.685732 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:27.685700 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:22:39.139874 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:22:39.139838 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:00.841550 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:00.841517 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:04.307780 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:04.307741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:05.453620 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:05.453583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:12.060297 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.060248 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:12.180195 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.180096 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:12.358339 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.358303 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb"] Sep 11 14:23:12.358553 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.358532 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" containerID="cri-o://d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17" gracePeriod=30 Sep 11 14:23:12.390761 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.390721 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:12.740243 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740210 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr"] Sep 11 14:23:12.740662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740648 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" Sep 11 14:23:12.740662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740664 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" Sep 11 14:23:12.740758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740672 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fe1636-5684-40a1-8a15-54685f37821c" containerName="payload-processing" Sep 11 14:23:12.740758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740678 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fe1636-5684-40a1-8a15-54685f37821c" containerName="payload-processing" Sep 11 14:23:12.740758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740693 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8" containerName="revoke-keys" Sep 11 14:23:12.740758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740698 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8" containerName="revoke-keys" Sep 11 14:23:12.740758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740709 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" Sep 11 14:23:12.740758 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740714 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" Sep 11 14:23:12.740970 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740799 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd2816b-b1d2-4b8b-adc3-3aff5272a6a8" containerName="revoke-keys" Sep 11 14:23:12.740970 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740807 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fe1636-5684-40a1-8a15-54685f37821c" containerName="payload-processing" Sep 11 14:23:12.740970 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740816 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" Sep 11 14:23:12.740970 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.740822 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="08225ba2-19d7-4234-91c6-3d2089aaa971" containerName="istio-proxy" Sep 11 14:23:12.744015 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.743999 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:12.748158 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.748136 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:23:12.748265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.748209 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:23:12.756949 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.756926 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr"] Sep 11 14:23:12.824244 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.824209 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172dc27c-55d1-437a-b9cf-4fa804f03c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr\" (UID: \"172dc27c-55d1-437a-b9cf-4fa804f03c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:12.925292 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.925253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172dc27c-55d1-437a-b9cf-4fa804f03c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr\" (UID: \"172dc27c-55d1-437a-b9cf-4fa804f03c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:12.925886 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.925868 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172dc27c-55d1-437a-b9cf-4fa804f03c57-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr\" (UID: \"172dc27c-55d1-437a-b9cf-4fa804f03c57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:12.989778 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:12.989741 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:13.054558 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:13.054533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:13.331960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:13.331933 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr"] Sep 11 14:23:13.332409 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:23:13.332290 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172dc27c_55d1_437a_b9cf_4fa804f03c57.slice/crio-1d212405c4f91fd05717b827c46eb2bcef42e1aec7c1128de1bd1e8b41cd8842 WatchSource:0}: Error finding container 1d212405c4f91fd05717b827c46eb2bcef42e1aec7c1128de1bd1e8b41cd8842: Status 404 returned error can't find the container with id 1d212405c4f91fd05717b827c46eb2bcef42e1aec7c1128de1bd1e8b41cd8842 Sep 11 14:23:14.055168 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:14.055125 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" event={"ID":"172dc27c-55d1-437a-b9cf-4fa804f03c57","Type":"ContainerStarted","Data":"24a9babc3118957ac1d3cd906c2a7cc8323e3dedc8e7b3eb730e36b686a1f890"} Sep 11 14:23:14.055363 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:14.055187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" event={"ID":"172dc27c-55d1-437a-b9cf-4fa804f03c57","Type":"ContainerStarted","Data":"1d212405c4f91fd05717b827c46eb2bcef42e1aec7c1128de1bd1e8b41cd8842"} Sep 11 14:23:14.094728 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:14.094683 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" podStartSLOduration=2.09466687 podStartE2EDuration="2.09466687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:23:14.093992271 +0000 UTC m=+1810.963723534" watchObservedRunningTime="2026-09-11 14:23:14.09466687 +0000 UTC m=+1810.964398131" Sep 11 14:23:14.265861 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:14.265613 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:15.063218 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:15.063180 2579 generic.go:356] "Generic (PLEG): container finished" podID="172dc27c-55d1-437a-b9cf-4fa804f03c57" containerID="24a9babc3118957ac1d3cd906c2a7cc8323e3dedc8e7b3eb730e36b686a1f890" exitCode=0 Sep 11 14:23:15.063615 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:15.063254 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" event={"ID":"172dc27c-55d1-437a-b9cf-4fa804f03c57","Type":"ContainerDied","Data":"24a9babc3118957ac1d3cd906c2a7cc8323e3dedc8e7b3eb730e36b686a1f890"} Sep 11 14:23:15.097763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:15.097725 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:15.966697 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:15.966654 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:16.222112 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:16.222025 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:16.362050 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:16.362016 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172dc27c-55d1-437a-b9cf-4fa804f03c57-maas-api-service-ca\") pod \"172dc27c-55d1-437a-b9cf-4fa804f03c57\" (UID: \"172dc27c-55d1-437a-b9cf-4fa804f03c57\") " Sep 11 14:23:16.362438 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:16.362405 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172dc27c-55d1-437a-b9cf-4fa804f03c57-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "172dc27c-55d1-437a-b9cf-4fa804f03c57" (UID: "172dc27c-55d1-437a-b9cf-4fa804f03c57"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:23:16.463412 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:16.463378 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/172dc27c-55d1-437a-b9cf-4fa804f03c57-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:16.836974 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:16.836935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:17.081750 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.081723 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" Sep 11 14:23:17.081939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.081759 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr" event={"ID":"172dc27c-55d1-437a-b9cf-4fa804f03c57","Type":"ContainerDied","Data":"1d212405c4f91fd05717b827c46eb2bcef42e1aec7c1128de1bd1e8b41cd8842"} Sep 11 14:23:17.081939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.081799 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d212405c4f91fd05717b827c46eb2bcef42e1aec7c1128de1bd1e8b41cd8842" Sep 11 14:23:17.416901 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.416869 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:17.621848 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.621783 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:23:17.776914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.776881 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istiod-ca-cert\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777094 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.776932 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-envoy\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777094 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.776965 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-certs\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777229 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777209 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z56ll\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-kube-api-access-z56ll\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777254 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-socket\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777280 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777277 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-data\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777381 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777287 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:17.777381 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777309 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-credential-socket\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777381 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777359 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:23:17.777536 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777428 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-token\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777536 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777481 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-podinfo\") pod \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\" (UID: \"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1\") " Sep 11 14:23:17.777649 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777532 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:17.777649 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777552 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:17.777726 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.777672 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-data" (OuterVolumeSpecName: "istio-data") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:17.778160 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.778138 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.778258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.778165 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.778258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.778179 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.778258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.778193 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.778258 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.778210 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.779511 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.779492 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:17.779585 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.779513 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-kube-api-access-z56ll" (OuterVolumeSpecName: "kube-api-access-z56ll") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "kube-api-access-z56ll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:17.780097 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.780074 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-token" (OuterVolumeSpecName: "istio-token") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:17.780097 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.780094 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" (UID: "ac1b6f52-816f-4d7f-8840-0a76bbccf1b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:23:17.879051 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.878966 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.879051 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.878996 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.879051 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.879004 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:17.879051 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:17.879013 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z56ll\" (UniqueName: \"kubernetes.io/projected/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1-kube-api-access-z56ll\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:18.090490 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.090456 2579 generic.go:356] "Generic (PLEG): container finished" podID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerID="d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17" exitCode=0 Sep 11 14:23:18.090662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.090501 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" Sep 11 14:23:18.090662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.090546 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" event={"ID":"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1","Type":"ContainerDied","Data":"d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17"} Sep 11 14:23:18.090662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.090604 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb" event={"ID":"ac1b6f52-816f-4d7f-8840-0a76bbccf1b1","Type":"ContainerDied","Data":"408bb8252065afabb1d0eb867c6b259df69cde798e002132b047ce64302c004d"} Sep 11 14:23:18.090662 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.090630 2579 scope.go:117] "RemoveContainer" containerID="d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17" Sep 11 14:23:18.101368 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.101350 2579 scope.go:117] "RemoveContainer" containerID="33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68" Sep 11 14:23:18.110868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.110849 2579 scope.go:117] "RemoveContainer" containerID="d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17" Sep 11 14:23:18.111218 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:23:18.111198 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17\": container with ID starting with d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17 not found: ID does not exist" containerID="d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17" Sep 11 14:23:18.111277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.111229 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17"} err="failed to get container status \"d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17\": rpc error: code = NotFound desc = could not find container \"d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17\": container with ID starting with d366265acd622cdda8e4d4e4ab391ad43e61bedc931148e58450dbe507ca4b17 not found: ID does not exist" Sep 11 14:23:18.111277 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.111247 2579 scope.go:117] "RemoveContainer" containerID="33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68" Sep 11 14:23:18.111502 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:23:18.111480 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68\": container with ID starting with 33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68 not found: ID does not exist" containerID="33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68" Sep 11 14:23:18.111540 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.111510 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68"} err="failed to get container status \"33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68\": rpc error: code = NotFound desc = could not find container \"33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68\": container with ID starting with 33dfe3b9ff103c3d45aa7d33b0f7937168639b7328aea217a6b922d2f1a14a68 not found: ID does not exist" Sep 11 14:23:18.118002 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.117975 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb"] Sep 11 14:23:18.122228 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:18.122202 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e6521d-openshift-default-5d6f56d69b-sjvrb"] Sep 11 14:23:19.643572 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:19.643541 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" path="/var/lib/kubelet/pods/ac1b6f52-816f-4d7f-8840-0a76bbccf1b1/volumes" Sep 11 14:23:23.329734 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.329703 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:23.430128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.430069 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:23.530128 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.530058 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:23.548986 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.548943 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr"] Sep 11 14:23:23.551592 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.551571 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d9cd47-zdkwr"] Sep 11 14:23:23.636526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.636437 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:23.643490 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.643464 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="172dc27c-55d1-437a-b9cf-4fa804f03c57" path="/var/lib/kubelet/pods/172dc27c-55d1-437a-b9cf-4fa804f03c57/volumes" Sep 11 14:23:23.929251 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:23.929145 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:24.036453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:24.036423 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:24.874952 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:24.874910 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:25.693483 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:25.693449 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:30.795596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:30.795557 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:30.894092 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:30.894045 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:31.090562 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.090468 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:31.416904 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.416809 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8"] Sep 11 14:23:31.417345 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417327 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" Sep 11 14:23:31.417449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417347 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" Sep 11 14:23:31.417449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417373 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="172dc27c-55d1-437a-b9cf-4fa804f03c57" containerName="revoke-keys" Sep 11 14:23:31.417449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417381 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="172dc27c-55d1-437a-b9cf-4fa804f03c57" containerName="revoke-keys" Sep 11 14:23:31.417449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417394 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" Sep 11 14:23:31.417449 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417403 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" Sep 11 14:23:31.417684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417538 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="172dc27c-55d1-437a-b9cf-4fa804f03c57" containerName="revoke-keys" Sep 11 14:23:31.417684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417550 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" Sep 11 14:23:31.417684 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.417559 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac1b6f52-816f-4d7f-8840-0a76bbccf1b1" containerName="istio-proxy" Sep 11 14:23:31.422224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.422204 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:31.428935 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.428911 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bwvnc\"" Sep 11 14:23:31.429451 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.429077 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:23:31.432192 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.432131 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8"] Sep 11 14:23:31.521204 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.521169 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b99f488-7e02-47b0-b047-be92c79d3112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8\" (UID: \"7b99f488-7e02-47b0-b047-be92c79d3112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:31.622780 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.622736 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b99f488-7e02-47b0-b047-be92c79d3112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8\" (UID: \"7b99f488-7e02-47b0-b047-be92c79d3112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:31.623400 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.623378 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b99f488-7e02-47b0-b047-be92c79d3112-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8\" (UID: \"7b99f488-7e02-47b0-b047-be92c79d3112\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:31.734799 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.734719 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:31.862804 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:31.862767 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:32.018080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:32.018058 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8"] Sep 11 14:23:32.020133 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:23:32.020082 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b99f488_7e02_47b0_b047_be92c79d3112.slice/crio-b3b70477647f79e5fc1d4b274c27da068d203bae68e88c08ee133d80ad4ba487 WatchSource:0}: Error finding container b3b70477647f79e5fc1d4b274c27da068d203bae68e88c08ee133d80ad4ba487: Status 404 returned error can't find the container with id b3b70477647f79e5fc1d4b274c27da068d203bae68e88c08ee133d80ad4ba487 Sep 11 14:23:32.189259 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:32.189229 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" event={"ID":"7b99f488-7e02-47b0-b047-be92c79d3112","Type":"ContainerStarted","Data":"b3b70477647f79e5fc1d4b274c27da068d203bae68e88c08ee133d80ad4ba487"} Sep 11 14:23:32.925074 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:32.925033 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:33.197691 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:33.197656 2579 generic.go:356] "Generic (PLEG): container finished" podID="7b99f488-7e02-47b0-b047-be92c79d3112" containerID="6b435a6c4e938cb64a388321e3087820745a8fc0d432233f172707138dbe4027" exitCode=0 Sep 11 14:23:33.197865 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:33.197741 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" event={"ID":"7b99f488-7e02-47b0-b047-be92c79d3112","Type":"ContainerDied","Data":"6b435a6c4e938cb64a388321e3087820745a8fc0d432233f172707138dbe4027"} Sep 11 14:23:33.833250 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:33.833215 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:34.351900 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:34.351871 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:34.450000 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:34.449967 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b99f488-7e02-47b0-b047-be92c79d3112-maas-api-service-ca\") pod \"7b99f488-7e02-47b0-b047-be92c79d3112\" (UID: \"7b99f488-7e02-47b0-b047-be92c79d3112\") " Sep 11 14:23:34.450364 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:34.450343 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b99f488-7e02-47b0-b047-be92c79d3112-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7b99f488-7e02-47b0-b047-be92c79d3112" (UID: "7b99f488-7e02-47b0-b047-be92c79d3112"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:23:34.551063 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:34.551039 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b99f488-7e02-47b0-b047-be92c79d3112-maas-api-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:34.750284 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:34.750241 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:35.213862 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:35.213831 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" Sep 11 14:23:35.214036 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:35.213835 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8" event={"ID":"7b99f488-7e02-47b0-b047-be92c79d3112","Type":"ContainerDied","Data":"b3b70477647f79e5fc1d4b274c27da068d203bae68e88c08ee133d80ad4ba487"} Sep 11 14:23:35.214036 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:35.213929 2579 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3b70477647f79e5fc1d4b274c27da068d203bae68e88c08ee133d80ad4ba487" Sep 11 14:23:42.129956 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.129920 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.138612 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.138584 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr"] Sep 11 14:23:42.141317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.141296 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fb65d4-29818935-pjkkr"] Sep 11 14:23:42.244728 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.244693 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.339021 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.338977 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.440849 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.440766 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8"] Sep 11 14:23:42.442870 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.442839 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.443492 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.443471 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-fb65d4-9tzp8"] Sep 11 14:23:42.812779 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.812739 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:42.969568 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:42.969528 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:43.650589 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:43.650556 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b99f488-7e02-47b0-b047-be92c79d3112" path="/var/lib/kubelet/pods/7b99f488-7e02-47b0-b047-be92c79d3112/volumes" Sep 11 14:23:43.650983 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:43.650825 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec718bd0-4d61-4e65-9440-d31ff3db3d96" path="/var/lib/kubelet/pods/ec718bd0-4d61-4e65-9440-d31ff3db3d96/volumes" Sep 11 14:23:44.009041 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:44.008937 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:49.367868 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:49.367836 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:49.492497 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:49.492455 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:49.688667 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:49.688580 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl"] Sep 11 14:23:49.688858 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:49.688832 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" containerID="cri-o://1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69" gracePeriod=30 Sep 11 14:23:49.725293 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:49.725244 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:50.383842 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:50.383802 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:51.240675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:51.240647 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:52.262950 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:52.262912 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:23:53.387680 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:53.387642 2579 patch_prober.go:28] interesting pod/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Sep 11 14:23:53.388062 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:53.387713 2579 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 11 14:23:54.934435 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:54.934406 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:23:55.051771 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051745 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-certs\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.051947 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051801 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-data\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.051947 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051825 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637c29a6-56e5-4dc2-b841-78d65be5961f-istiod-ca-cert\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.051947 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051860 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-token\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.051947 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051919 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flkkx\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-kube-api-access-flkkx\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.052275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051965 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-envoy\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.052275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.051986 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-podinfo\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.052275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052011 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-socket\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.052275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052014 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:55.052275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052033 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-credential-socket\") pod \"637c29a6-56e5-4dc2-b841-78d65be5961f\" (UID: \"637c29a6-56e5-4dc2-b841-78d65be5961f\") " Sep 11 14:23:55.052275 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052139 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-data" (OuterVolumeSpecName: "istio-data") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:55.052595 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052309 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/637c29a6-56e5-4dc2-b841-78d65be5961f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:23:55.052595 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052404 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.052595 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052425 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.052595 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052401 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:55.052970 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.052699 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:55.054458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.054432 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:23:55.054616 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.054496 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-kube-api-access-flkkx" (OuterVolumeSpecName: "kube-api-access-flkkx") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "kube-api-access-flkkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:55.054903 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.054878 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:23:55.055159 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.055130 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-token" (OuterVolumeSpecName: "istio-token") pod "637c29a6-56e5-4dc2-b841-78d65be5961f" (UID: "637c29a6-56e5-4dc2-b841-78d65be5961f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:23:55.153453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153417 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/637c29a6-56e5-4dc2-b841-78d65be5961f-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.153453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153445 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.153453 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153456 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flkkx\" (UniqueName: \"kubernetes.io/projected/637c29a6-56e5-4dc2-b841-78d65be5961f-kube-api-access-flkkx\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.153685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153465 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.153685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153473 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/637c29a6-56e5-4dc2-b841-78d65be5961f-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.153685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153481 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.153685 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.153489 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/637c29a6-56e5-4dc2-b841-78d65be5961f-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:23:55.342914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.342824 2579 generic.go:356] "Generic (PLEG): container finished" podID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerID="1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69" exitCode=0 Sep 11 14:23:55.342914 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.342873 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" Sep 11 14:23:55.343119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.342906 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" event={"ID":"637c29a6-56e5-4dc2-b841-78d65be5961f","Type":"ContainerDied","Data":"1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69"} Sep 11 14:23:55.343119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.342966 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl" event={"ID":"637c29a6-56e5-4dc2-b841-78d65be5961f","Type":"ContainerDied","Data":"a5d21d910d14a2250235f224bc91243a4ae904b7a055f337db195268ad391862"} Sep 11 14:23:55.343119 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.342987 2579 scope.go:117] "RemoveContainer" containerID="1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69" Sep 11 14:23:55.352968 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.352952 2579 scope.go:117] "RemoveContainer" containerID="55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda" Sep 11 14:23:55.361680 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.361665 2579 scope.go:117] "RemoveContainer" containerID="1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69" Sep 11 14:23:55.361935 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:23:55.361913 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69\": container with ID starting with 1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69 not found: ID does not exist" containerID="1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69" Sep 11 14:23:55.362001 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.361943 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69"} err="failed to get container status \"1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69\": rpc error: code = NotFound desc = could not find container \"1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69\": container with ID starting with 1b94c3e30c53821fca5c8f92ee6ef41873e679e5989872cdca5746db5fbb1b69 not found: ID does not exist" Sep 11 14:23:55.362001 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.361960 2579 scope.go:117] "RemoveContainer" containerID="55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda" Sep 11 14:23:55.362252 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:23:55.362233 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda\": container with ID starting with 55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda not found: ID does not exist" containerID="55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda" Sep 11 14:23:55.362303 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.362257 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda"} err="failed to get container status \"55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda\": rpc error: code = NotFound desc = could not find container \"55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda\": container with ID starting with 55d9e2470b4a1280bfcb4f520e941121f98fa948d967fa75017dcc474a33bbda not found: ID does not exist" Sep 11 14:23:55.369458 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.369435 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl"] Sep 11 14:23:55.375973 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.375953 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-fb65d4-openshift-default-598f9df9cb-pfwxl"] Sep 11 14:23:55.643237 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:23:55.643163 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" path="/var/lib/kubelet/pods/637c29a6-56e5-4dc2-b841-78d65be5961f/volumes" Sep 11 14:24:00.668790 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:00.668758 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:00.681810 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:00.681777 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t"] Sep 11 14:24:00.685997 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:00.685976 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-261f9c-29818935-2568t"] Sep 11 14:24:00.769155 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:00.769124 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:00.869339 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:00.869311 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:00.949653 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:00.949566 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:01.316733 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:01.316701 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:01.418831 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:01.418793 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:01.642814 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:01.642719 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d70d6c-5209-43e0-a363-7f18858d7bfb" path="/var/lib/kubelet/pods/97d70d6c-5209-43e0-a363-7f18858d7bfb/volumes" Sep 11 14:24:04.103501 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:04.103469 2579 scope.go:117] "RemoveContainer" containerID="3b8cca7d7394b0679c36a91d0cec2ca2bbb134261f4fb864282734d1c952124d" Sep 11 14:24:04.113115 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:04.113078 2579 scope.go:117] "RemoveContainer" containerID="9ddf40d7e7d8b3fc2a1998002affec7f1d88706bb910ddd6a91fd5249e25bb45" Sep 11 14:24:08.387445 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:08.387404 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:08.472509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:08.472461 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:08.573465 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:08.573435 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k"] Sep 11 14:24:08.573702 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:08.573675 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" containerID="cri-o://bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36" gracePeriod=30 Sep 11 14:24:08.597749 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:08.597717 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:24:13.822431 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.822404 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:24:13.934383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934290 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-envoy\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934334 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6012c421-2586-43f1-b418-0dcb790731bb-istiod-ca-cert\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934350 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-socket\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934392 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-certs\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934413 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-credential-socket\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934437 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9hfw\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-kube-api-access-x9hfw\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934469 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6012c421-2586-43f1-b418-0dcb790731bb-istio-podinfo\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934501 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-data\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934676 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934570 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-istio-token\") pod \"6012c421-2586-43f1-b418-0dcb790731bb\" (UID: \"6012c421-2586-43f1-b418-0dcb790731bb\") " Sep 11 14:24:13.934972 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934751 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:24:13.934972 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934768 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6012c421-2586-43f1-b418-0dcb790731bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:24:13.935091 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934972 2579 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:13.935091 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934987 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:24:13.935091 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.934991 2579 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6012c421-2586-43f1-b418-0dcb790731bb-istiod-ca-cert\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:13.935298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.935189 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:24:13.935517 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.935341 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:24:13.936850 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.936814 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 11 14:24:13.936850 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.936820 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6012c421-2586-43f1-b418-0dcb790731bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Sep 11 14:24:13.937044 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.936937 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:24:13.937276 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:13.937253 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-kube-api-access-x9hfw" (OuterVolumeSpecName: "kube-api-access-x9hfw") pod "6012c421-2586-43f1-b418-0dcb790731bb" (UID: "6012c421-2586-43f1-b418-0dcb790731bb"). InnerVolumeSpecName "kube-api-access-x9hfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:24:14.035512 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035473 2579 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-istio-token\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.035512 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035506 2579 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-envoy\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.035512 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035516 2579 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-workload-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.035763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035525 2579 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-credential-socket\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.035763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035535 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9hfw\" (UniqueName: \"kubernetes.io/projected/6012c421-2586-43f1-b418-0dcb790731bb-kube-api-access-x9hfw\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.035763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035545 2579 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6012c421-2586-43f1-b418-0dcb790731bb-istio-podinfo\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.035763 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.035554 2579 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6012c421-2586-43f1-b418-0dcb790731bb-istio-data\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:24:14.465143 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.465083 2579 generic.go:356] "Generic (PLEG): container finished" podID="6012c421-2586-43f1-b418-0dcb790731bb" containerID="bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36" exitCode=0 Sep 11 14:24:14.465342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.465171 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" event={"ID":"6012c421-2586-43f1-b418-0dcb790731bb","Type":"ContainerDied","Data":"bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36"} Sep 11 14:24:14.465342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.465252 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" event={"ID":"6012c421-2586-43f1-b418-0dcb790731bb","Type":"ContainerDied","Data":"85dd2a3f8f785fb18714be52edfcc5c91d5cb7d7f7865e388e5efb79e3f38164"} Sep 11 14:24:14.465342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.465179 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k" Sep 11 14:24:14.465342 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.465274 2579 scope.go:117] "RemoveContainer" containerID="bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36" Sep 11 14:24:14.475951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.475932 2579 scope.go:117] "RemoveContainer" containerID="5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e" Sep 11 14:24:14.484918 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.484896 2579 scope.go:117] "RemoveContainer" containerID="bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36" Sep 11 14:24:14.485254 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:24:14.485232 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36\": container with ID starting with bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36 not found: ID does not exist" containerID="bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36" Sep 11 14:24:14.485354 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.485264 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36"} err="failed to get container status \"bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36\": rpc error: code = NotFound desc = could not find container \"bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36\": container with ID starting with bb5d65a4729320452556e5e7b054696bd598e7b359a9f549e0906d0766d9fa36 not found: ID does not exist" Sep 11 14:24:14.485354 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.485283 2579 scope.go:117] "RemoveContainer" containerID="5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e" Sep 11 14:24:14.485531 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:24:14.485514 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e\": container with ID starting with 5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e not found: ID does not exist" containerID="5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e" Sep 11 14:24:14.485600 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.485533 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e"} err="failed to get container status \"5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e\": rpc error: code = NotFound desc = could not find container \"5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e\": container with ID starting with 5ef794e673523d9b8f9898daf53e0f3001c7d0538ffc2bded95ddc3849280f1e not found: ID does not exist" Sep 11 14:24:14.495255 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.495214 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k"] Sep 11 14:24:14.498057 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:14.498030 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-261f9c-openshift-default-6676676dcd-rpc8k"] Sep 11 14:24:15.643880 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:15.643847 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6012c421-2586-43f1-b418-0dcb790731bb" path="/var/lib/kubelet/pods/6012c421-2586-43f1-b418-0dcb790731bb/volumes" Sep 11 14:24:52.293059 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:24:52.292985 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:04.148728 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:25:04.148698 2579 scope.go:117] "RemoveContainer" containerID="2cd6a8f9c813cfa3f96a4f4703a8b43980e9ddc4b55d0833bfc38d9fbac1482c" Sep 11 14:25:33.284450 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:25:33.284414 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:25:34.999967 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:25:34.999932 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:04.173762 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:26:04.173724 2579 scope.go:117] "RemoveContainer" containerID="2c53799adb955526d02f58e19cdb9db80a060655ddefbce292c4e609e55d6ffe" Sep 11 14:26:04.182872 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:26:04.182852 2579 scope.go:117] "RemoveContainer" containerID="032c479ac658ba54f2b4b9ba7a4b8a368eab66ec62b7e60d6bc0d5fd92276bbb" Sep 11 14:26:17.617980 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:26:17.617935 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:26:38.446073 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:26:38.446016 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:27:17.553464 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:27:17.553431 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:04.217770 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:28:04.217732 2579 scope.go:117] "RemoveContainer" containerID="ee4720eb6abc4905f631df5143212d878ddb04de18ec1134be1da1543e971931" Sep 11 14:28:45.513939 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:28:45.513893 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:28:55.842162 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:28:55.842067 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:04.239006 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:04.238975 2579 scope.go:117] "RemoveContainer" containerID="290406f5a23fd6af96c155a13e8be71576fea733778202e7bfdc085e01078bb2" Sep 11 14:29:36.458549 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:36.458512 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:37.421941 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:37.421914 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:29:57.557977 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.557943 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4"] Sep 11 14:29:57.558412 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558398 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" Sep 11 14:29:57.558412 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558413 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" Sep 11 14:29:57.558526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558428 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" Sep 11 14:29:57.558526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558434 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" Sep 11 14:29:57.558526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558441 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b99f488-7e02-47b0-b047-be92c79d3112" containerName="revoke-keys" Sep 11 14:29:57.558526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558447 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b99f488-7e02-47b0-b047-be92c79d3112" containerName="revoke-keys" Sep 11 14:29:57.558526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558461 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" Sep 11 14:29:57.558526 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558469 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" Sep 11 14:29:57.558722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558545 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" Sep 11 14:29:57.558722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558555 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="637c29a6-56e5-4dc2-b841-78d65be5961f" containerName="istio-proxy" Sep 11 14:29:57.558722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558564 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b99f488-7e02-47b0-b047-be92c79d3112" containerName="revoke-keys" Sep 11 14:29:57.558722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558571 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" Sep 11 14:29:57.558722 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.558577 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" Sep 11 14:29:57.561650 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.561632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.565984 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.565961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-metrics-certs\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.566197 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.566019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mjdd\" (UniqueName: \"kubernetes.io/projected/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-kube-api-access-5mjdd\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.566197 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.566043 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-openshift-service-ca\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.566197 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.566070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-maas-api-tls\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.567520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.567492 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Sep 11 14:29:57.567520 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.567514 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Sep 11 14:29:57.567675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.567514 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-cmhl6\"" Sep 11 14:29:57.567675 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.567497 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Sep 11 14:29:57.567949 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.567931 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Sep 11 14:29:57.595210 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.595183 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4"] Sep 11 14:29:57.667227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.667190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-maas-api-tls\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.667428 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.667305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-metrics-certs\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.667428 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.667369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mjdd\" (UniqueName: \"kubernetes.io/projected/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-kube-api-access-5mjdd\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.667428 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.667399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-openshift-service-ca\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.668139 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.668052 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-openshift-service-ca\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.669824 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.669804 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-maas-api-tls\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.669923 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.669845 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-metrics-certs\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.697022 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.696990 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mjdd\" (UniqueName: \"kubernetes.io/projected/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-kube-api-access-5mjdd\") pod \"maas-api-56ffc8465-tvdn4\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:57.872080 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:57.871995 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:29:58.159077 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:58.159047 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4"] Sep 11 14:29:58.159985 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:29:58.159960 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c78db6e_24c8_465b_b0ad_ffe5897e2ae3.slice/crio-2f702f2e518d990dd623ddf7c76a2272035def8790b6cf0bda49d98d6ce48085 WatchSource:0}: Error finding container 2f702f2e518d990dd623ddf7c76a2272035def8790b6cf0bda49d98d6ce48085: Status 404 returned error can't find the container with id 2f702f2e518d990dd623ddf7c76a2272035def8790b6cf0bda49d98d6ce48085 Sep 11 14:29:58.161396 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:58.161376 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 11 14:29:58.530618 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:29:58.530532 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" event={"ID":"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3","Type":"ContainerStarted","Data":"2f702f2e518d990dd623ddf7c76a2272035def8790b6cf0bda49d98d6ce48085"} Sep 11 14:30:01.551729 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:01.551687 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" event={"ID":"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3","Type":"ContainerStarted","Data":"a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7"} Sep 11 14:30:01.552143 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:01.551913 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:30:01.577522 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:01.577475 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" podStartSLOduration=2.214949919 podStartE2EDuration="4.57745942s" totalImagesPullingTime="2.362509501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 14:29:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:01.575273808 +0000 UTC m=+2218.445005082" watchObservedRunningTime="2026-09-11 14:30:01.57745942 +0000 UTC m=+2218.447190681" Sep 11 14:30:04.262529 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:04.262498 2579 scope.go:117] "RemoveContainer" containerID="6b435a6c4e938cb64a388321e3087820745a8fc0d432233f172707138dbe4027" Sep 11 14:30:04.271805 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:04.271783 2579 scope.go:117] "RemoveContainer" containerID="24a9babc3118957ac1d3cd906c2a7cc8323e3dedc8e7b3eb730e36b686a1f890" Sep 11 14:30:05.708746 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:05.708706 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4"] Sep 11 14:30:05.709180 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:05.708970 2579 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" podUID="9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" containerName="maas-api" containerID="cri-o://a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7" gracePeriod=30 Sep 11 14:30:05.714429 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:05.714407 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:30:05.968003 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:05.967942 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:30:06.055665 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.055626 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mjdd\" (UniqueName: \"kubernetes.io/projected/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-kube-api-access-5mjdd\") pod \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " Sep 11 14:30:06.055926 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.055750 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-openshift-service-ca\") pod \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " Sep 11 14:30:06.055926 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.055789 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-maas-api-tls\") pod \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " Sep 11 14:30:06.055926 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.055854 2579 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-metrics-certs\") pod \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\" (UID: \"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3\") " Sep 11 14:30:06.056157 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.056128 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" (UID: "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 11 14:30:06.058118 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.058070 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" (UID: "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:30:06.058227 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.058191 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" (UID: "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 11 14:30:06.058265 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.058220 2579 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-kube-api-access-5mjdd" (OuterVolumeSpecName: "kube-api-access-5mjdd") pod "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" (UID: "9c78db6e-24c8-465b-b0ad-ffe5897e2ae3"). InnerVolumeSpecName "kube-api-access-5mjdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 11 14:30:06.157485 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.157443 2579 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-openshift-service-ca\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:30:06.157485 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.157476 2579 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-maas-api-tls\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:30:06.157485 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.157485 2579 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-metrics-certs\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:30:06.157485 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.157493 2579 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mjdd\" (UniqueName: \"kubernetes.io/projected/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3-kube-api-access-5mjdd\") on node \"ip-10-0-136-162.ec2.internal\" DevicePath \"\"" Sep 11 14:30:06.582781 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.582740 2579 generic.go:356] "Generic (PLEG): container finished" podID="9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" containerID="a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7" exitCode=0 Sep 11 14:30:06.582976 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.582789 2579 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" Sep 11 14:30:06.582976 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.582821 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" event={"ID":"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3","Type":"ContainerDied","Data":"a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7"} Sep 11 14:30:06.582976 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.582875 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4" event={"ID":"9c78db6e-24c8-465b-b0ad-ffe5897e2ae3","Type":"ContainerDied","Data":"2f702f2e518d990dd623ddf7c76a2272035def8790b6cf0bda49d98d6ce48085"} Sep 11 14:30:06.582976 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.582897 2579 scope.go:117] "RemoveContainer" containerID="a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7" Sep 11 14:30:06.593126 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.593080 2579 scope.go:117] "RemoveContainer" containerID="a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7" Sep 11 14:30:06.593457 ip-10-0-136-162 kubenswrapper[2579]: E0911 14:30:06.593438 2579 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7\": container with ID starting with a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7 not found: ID does not exist" containerID="a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7" Sep 11 14:30:06.593534 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.593469 2579 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7"} err="failed to get container status \"a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7\": rpc error: code = NotFound desc = could not find container \"a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7\": container with ID starting with a7b8fdc707589799a3abffa2d687532013a3d29723061af56e403f08fe4158e7 not found: ID does not exist" Sep 11 14:30:06.615415 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.615378 2579 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4"] Sep 11 14:30:06.622704 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:06.622679 2579 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-56ffc8465-tvdn4"] Sep 11 14:30:07.643714 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:07.643678 2579 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" path="/var/lib/kubelet/pods/9c78db6e-24c8-465b-b0ad-ffe5897e2ae3/volumes" Sep 11 14:30:11.764310 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:11.764275 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:33.562457 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:33.562370 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7zncc_1241f51c-d8ea-4b01-8784-1f88bd5be660/manager/0.log" Sep 11 14:30:33.973177 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:33.973076 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-tmbdb_c3787eaa-67b6-4214-8889-9c6aa4a15c6b/copy-manifests/0.log" Sep 11 14:30:33.983949 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:33.983929 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-tmbdb_c3787eaa-67b6-4214-8889-9c6aa4a15c6b/manager/0.log" Sep 11 14:30:35.651125 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:35.651074 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-2cnmk_59493059-1289-4d1c-99b9-d4b3d1603d61/manager/0.log" Sep 11 14:30:36.241393 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:36.241363 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-99d59_8ce4f934-4c13-4fef-a4af-4136e1038b1b/manager/0.log" Sep 11 14:30:36.601198 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:36.601170 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5_c7405e28-03ad-4b1f-b1b8-a120c901998c/istio-proxy/0.log" Sep 11 14:30:36.726281 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:36.726251 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j97pv_79e32aa0-f398-4614-a389-7ce7913086eb/discovery/0.log" Sep 11 14:30:36.945535 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:36.945452 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-js2tp_f3137645-bb52-4763-905c-70b1f787077f/kube-auth-proxy/0.log" Sep 11 14:30:37.068183 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:37.068148 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/1.log" Sep 11 14:30:37.194289 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:37.193901 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fzzg5_06f0606d-a1dc-43d2-9264-7e2c583ce576/payload-pre-processing/0.log" Sep 11 14:30:37.779714 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:37.779677 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-gkzzl_7c1fe7ec-06f7-47cc-90fa-b233c76d5f3e/main/0.log" Sep 11 14:30:38.016571 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:38.016541 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-m2bvp_6d334514-92f2-4d83-8791-4892bf569912/main/0.log" Sep 11 14:30:38.137754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:38.137716 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-s9996_172bdcd6-0212-4e76-b7ec-846d1fcf0313/main/0.log" Sep 11 14:30:44.290516 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:44.290466 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:44.398121 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:44.398074 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" limit=1024 Sep 11 14:30:47.724539 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:47.724500 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-nxrhp_5b3eaae6-ad54-476c-8449-f8b9ab33a196/global-pull-secret-syncer/0.log" Sep 11 14:30:47.826932 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:47.826899 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-2dk4m_96df2f98-53ac-4384-b273-e57016a3ec89/konnectivity-agent/0.log" Sep 11 14:30:48.044902 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:48.044864 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-162.ec2.internal_1a570e7183fc751dd6686e350e76db83/haproxy/0.log" Sep 11 14:30:52.986371 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:52.986340 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-2cnmk_59493059-1289-4d1c-99b9-d4b3d1603d61/manager/0.log" Sep 11 14:30:53.238160 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:53.238056 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-99d59_8ce4f934-4c13-4fef-a4af-4136e1038b1b/manager/0.log" Sep 11 14:30:54.718826 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.718797 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/alertmanager/0.log" Sep 11 14:30:54.753068 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.753034 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/config-reloader/0.log" Sep 11 14:30:54.786648 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.786622 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/kube-rbac-proxy-web/0.log" Sep 11 14:30:54.818224 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.818193 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/kube-rbac-proxy/0.log" Sep 11 14:30:54.864456 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.864428 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/kube-rbac-proxy-metric/0.log" Sep 11 14:30:54.896971 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.896945 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/prom-label-proxy/0.log" Sep 11 14:30:54.933978 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:54.933957 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_cf7a111e-28f9-4f43-9089-a93bbd485e25/init-config-reloader/0.log" Sep 11 14:30:55.043585 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.043513 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-7ljt8_2bc881d3-aa82-48b9-a847-dc3c171d514c/kube-state-metrics/0.log" Sep 11 14:30:55.094515 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.094487 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-7ljt8_2bc881d3-aa82-48b9-a847-dc3c171d514c/kube-rbac-proxy-main/0.log" Sep 11 14:30:55.140754 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.140717 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-59fbdd77d-7ljt8_2bc881d3-aa82-48b9-a847-dc3c171d514c/kube-rbac-proxy-self/0.log" Sep 11 14:30:55.225838 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.225816 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-5d85778b9d-8kfw9_95aff6cc-a715-473c-8e92-9303dc3539b4/monitoring-plugin/0.log" Sep 11 14:30:55.383960 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.383936 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vk9hj_9a33cad5-3e43-4d5f-9df0-227a64c286b3/node-exporter/0.log" Sep 11 14:30:55.414239 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.414213 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vk9hj_9a33cad5-3e43-4d5f-9df0-227a64c286b3/kube-rbac-proxy/0.log" Sep 11 14:30:55.443109 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.443087 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vk9hj_9a33cad5-3e43-4d5f-9df0-227a64c286b3/init-textfile/0.log" Sep 11 14:30:55.942596 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.942573 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-zpcgl_38cabd2b-39bc-4cfb-a243-ef957d6d825c/prometheus-operator/0.log" Sep 11 14:30:55.977977 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:55.977948 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-68bcd7f987-zpcgl_38cabd2b-39bc-4cfb-a243-ef957d6d825c/kube-rbac-proxy/0.log" Sep 11 14:30:56.062157 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:56.062125 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65956475c7-xqv99_e1746003-2067-4c36-9dcf-f9bcb4eca44a/telemeter-client/0.log" Sep 11 14:30:56.092736 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:56.092709 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65956475c7-xqv99_e1746003-2067-4c36-9dcf-f9bcb4eca44a/reload/0.log" Sep 11 14:30:56.131456 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:56.131388 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-65956475c7-xqv99_e1746003-2067-4c36-9dcf-f9bcb4eca44a/kube-rbac-proxy/0.log" Sep 11 14:30:57.160829 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.160793 2579 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh"] Sep 11 14:30:57.161243 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.161230 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" Sep 11 14:30:57.161243 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.161244 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="6012c421-2586-43f1-b418-0dcb790731bb" containerName="istio-proxy" Sep 11 14:30:57.161317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.161259 2579 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" containerName="maas-api" Sep 11 14:30:57.161317 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.161265 2579 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" containerName="maas-api" Sep 11 14:30:57.161384 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.161337 2579 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c78db6e-24c8-465b-b0ad-ffe5897e2ae3" containerName="maas-api" Sep 11 14:30:57.163373 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.163357 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.166046 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.166023 2579 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5cfjv\"/\"default-dockercfg-jjc6s\"" Sep 11 14:30:57.166189 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.166092 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5cfjv\"/\"openshift-service-ca.crt\"" Sep 11 14:30:57.166697 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.166681 2579 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5cfjv\"/\"kube-root-ca.crt\"" Sep 11 14:30:57.182385 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.182359 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh"] Sep 11 14:30:57.250359 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.250331 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-podres\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.250503 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.250365 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwrvr\" (UniqueName: \"kubernetes.io/projected/03a85a91-57b6-466c-b9a4-a2fc3b492212-kube-api-access-fwrvr\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.250503 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.250397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-proc\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.250575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.250494 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-sys\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.250575 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.250538 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-lib-modules\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-podres\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351395 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351399 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwrvr\" (UniqueName: \"kubernetes.io/projected/03a85a91-57b6-466c-b9a4-a2fc3b492212-kube-api-access-fwrvr\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351426 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-proc\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351471 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-sys\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-lib-modules\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351555 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-podres\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351574 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-proc\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351605 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351588 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-sys\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.351819 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.351622 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/03a85a91-57b6-466c-b9a4-a2fc3b492212-lib-modules\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.362706 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.362683 2579 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwrvr\" (UniqueName: \"kubernetes.io/projected/03a85a91-57b6-466c-b9a4-a2fc3b492212-kube-api-access-fwrvr\") pod \"perf-node-gather-daemonset-rjfqh\" (UID: \"03a85a91-57b6-466c-b9a4-a2fc3b492212\") " pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.473632 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.473552 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.681739 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.681712 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-4sfhz_d6bf66f8-5d35-44c4-ba30-4d52c5519a22/networking-console-plugin/0.log" Sep 11 14:30:57.754023 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.754001 2579 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh"] Sep 11 14:30:57.756031 ip-10-0-136-162 kubenswrapper[2579]: W0911 14:30:57.756006 2579 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod03a85a91_57b6_466c_b9a4_a2fc3b492212.slice/crio-f486bb9669642bdd7594b8629c62202a4795fe446340e724b1a070ce0d51b83a WatchSource:0}: Error finding container f486bb9669642bdd7594b8629c62202a4795fe446340e724b1a070ce0d51b83a: Status 404 returned error can't find the container with id f486bb9669642bdd7594b8629c62202a4795fe446340e724b1a070ce0d51b83a Sep 11 14:30:57.888164 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.888136 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" event={"ID":"03a85a91-57b6-466c-b9a4-a2fc3b492212","Type":"ContainerStarted","Data":"ed54842f6587965f52aa4b1a1c2f5a737afc838205f854b45e4beeb0c449adc9"} Sep 11 14:30:57.888272 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.888187 2579 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" event={"ID":"03a85a91-57b6-466c-b9a4-a2fc3b492212","Type":"ContainerStarted","Data":"f486bb9669642bdd7594b8629c62202a4795fe446340e724b1a070ce0d51b83a"} Sep 11 14:30:57.888272 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.888215 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:30:57.916509 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:57.916462 2579 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" podStartSLOduration=0.916446276 podStartE2EDuration="916.446276ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 14:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 14:30:57.913793163 +0000 UTC m=+2274.783524424" watchObservedRunningTime="2026-09-11 14:30:57.916446276 +0000 UTC m=+2274.786177539" Sep 11 14:30:59.127263 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:59.127233 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-9cf8c5469-wg6rb_a9a68ec6-3272-4926-97cb-8a59561dd60f/console/0.log" Sep 11 14:30:59.174622 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:59.174586 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-76f99584f4-4pmwh_ae9357c0-5a13-4f62-85f8-8cda41eb985c/download-server/0.log" Sep 11 14:30:59.900202 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:30:59.900163 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-86695b7cc7-ljsn4_10d8c0d5-cd9a-4ff7-a719-ea1ac937a986/volume-data-source-validator/0.log" Sep 11 14:31:01.270585 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:01.270549 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g8b9r_429e5452-0512-4330-8cf1-4581154d1055/dns/0.log" Sep 11 14:31:01.332377 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:01.332349 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-g8b9r_429e5452-0512-4330-8cf1-4581154d1055/kube-rbac-proxy/0.log" Sep 11 14:31:01.583480 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:01.583452 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-kkhl9_06d275ee-fd61-4460-8bb1-013e6f6d08d2/dns-node-resolver/0.log" Sep 11 14:31:02.282645 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:02.282614 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-797f7fc5b4-852mw_d18dcec8-7619-4a8a-b790-15564fb955dc/registry/0.log" Sep 11 14:31:02.403646 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:02.403621 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jttfb_9036a4b6-21fd-47ea-9092-556af9ba0a7b/node-ca/0.log" Sep 11 14:31:03.721154 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:03.721121 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5_c7405e28-03ad-4b1f-b1b8-a120c901998c/istio-proxy/0.log" Sep 11 14:31:03.906626 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:03.906572 2579 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5cfjv/perf-node-gather-daemonset-rjfqh" Sep 11 14:31:04.287809 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:04.287778 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j97pv_79e32aa0-f398-4614-a389-7ce7913086eb/discovery/0.log" Sep 11 14:31:04.424787 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:04.424757 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-js2tp_f3137645-bb52-4763-905c-70b1f787077f/kube-auth-proxy/0.log" Sep 11 14:31:04.475559 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:04.475529 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/0.log" Sep 11 14:31:04.491170 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:04.491137 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/1.log" Sep 11 14:31:04.542155 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:04.542058 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fzzg5_06f0606d-a1dc-43d2-9264-7e2c583ce576/payload-pre-processing/0.log" Sep 11 14:31:05.712554 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:05.712526 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jgljh_596d001e-42a1-409b-95dd-71995ca53f2e/serve-healthcheck-canary/0.log" Sep 11 14:31:06.778164 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:06.778138 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2z2k_e3b0b9eb-9b0c-4839-a125-edeb7922bc51/kube-rbac-proxy/0.log" Sep 11 14:31:06.817787 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:06.817752 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2z2k_e3b0b9eb-9b0c-4839-a125-edeb7922bc51/exporter/0.log" Sep 11 14:31:06.867383 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:06.867358 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-s2z2k_e3b0b9eb-9b0c-4839-a125-edeb7922bc51/extractor/0.log" Sep 11 14:31:08.151298 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:08.151267 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5_c7405e28-03ad-4b1f-b1b8-a120c901998c/istio-proxy/0.log" Sep 11 14:31:08.615653 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:08.615623 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j97pv_79e32aa0-f398-4614-a389-7ce7913086eb/discovery/0.log" Sep 11 14:31:08.719232 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:08.719203 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-js2tp_f3137645-bb52-4763-905c-70b1f787077f/kube-auth-proxy/0.log" Sep 11 14:31:08.755951 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:08.755877 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/0.log" Sep 11 14:31:08.772047 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:08.772021 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/1.log" Sep 11 14:31:08.830203 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:08.830176 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fzzg5_06f0606d-a1dc-43d2-9264-7e2c583ce576/payload-pre-processing/0.log" Sep 11 14:31:10.175732 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:10.175683 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-7zncc_1241f51c-d8ea-4b01-8784-1f88bd5be660/manager/0.log" Sep 11 14:31:10.400615 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:10.400585 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-p8nh5_c7405e28-03ad-4b1f-b1b8-a120c901998c/istio-proxy/0.log" Sep 11 14:31:10.855358 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:10.855330 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-j97pv_79e32aa0-f398-4614-a389-7ce7913086eb/discovery/0.log" Sep 11 14:31:10.932664 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:10.932617 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-tmbdb_c3787eaa-67b6-4214-8889-9c6aa4a15c6b/manager/0.log" Sep 11 14:31:11.018963 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:11.018938 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7ff676d985-tmbdb_c3787eaa-67b6-4214-8889-9c6aa4a15c6b/copy-manifests/0.log" Sep 11 14:31:11.046586 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:11.046560 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-746c8b5fff-js2tp_f3137645-bb52-4763-905c-70b1f787077f/kube-auth-proxy/0.log" Sep 11 14:31:11.097892 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:11.097865 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/0.log" Sep 11 14:31:11.111944 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:11.111877 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-vlz4j_d91cba3f-bcf1-42cc-921f-f41d9be3f014/istio-proxy/1.log" Sep 11 14:31:11.192891 ip-10-0-136-162 kubenswrapper[2579]: I0911 14:31:11.192864 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-fzzg5_06f0606d-a1dc-43d2-9264-7e2c583ce576/payload-pre-processing/0.log"