Jul 28 17:55:02.370863 ip-10-0-135-147 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 17:55:02.370875 ip-10-0-135-147 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 17:55:02.370882 ip-10-0-135-147 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 17:55:02.371109 ip-10-0-135-147 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 17:55:12.934351 ip-10-0-135-147 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 28 17:55:12.934365 ip-10-0-135-147 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot a48c644fc3294d4db38606d26e079f26 -- Jul 28 17:56:42.468006 ip-10-0-135-147 systemd[1]: Starting Kubernetes Kubelet... Jul 28 17:56:42.868337 ip-10-0-135-147 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:56:42.868337 ip-10-0-135-147 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 17:56:42.868337 ip-10-0-135-147 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:56:42.868337 ip-10-0-135-147 kubenswrapper[2575]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 17:56:42.868337 ip-10-0-135-147 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 17:56:42.869092 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.868411 2575 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 17:56:42.874374 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874345 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:56:42.874374 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874370 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:56:42.874374 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874375 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:56:42.874374 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874379 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874382 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874385 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874391 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874394 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874397 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874399 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874402 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874405 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874408 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874411 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874414 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874416 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874419 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874422 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874425 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874428 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874430 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874433 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:56:42.874509 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874437 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874441 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874444 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874447 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874451 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874455 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874458 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874461 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874464 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874467 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874470 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874473 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874475 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874478 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874480 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874484 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874486 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874491 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874493 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874496 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:56:42.875022 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874499 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874501 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874505 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874507 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874510 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874512 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874515 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874518 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874521 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874523 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874526 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874528 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874532 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874535 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874537 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874540 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874543 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874545 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874548 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874551 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:56:42.875502 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874555 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874559 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874563 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874565 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874568 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874571 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874573 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874577 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874580 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874582 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874585 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874588 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874591 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874596 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874599 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874602 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874604 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874607 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874609 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874612 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:56:42.876014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874614 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874617 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874620 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874622 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874625 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874628 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874630 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874633 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874635 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874638 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874641 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874644 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874646 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874649 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874652 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874654 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.874657 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875453 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875460 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875467 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:56:42.876492 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875471 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875473 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875476 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875479 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875481 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875484 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875491 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875495 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875498 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875502 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875505 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875508 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875511 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875514 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875516 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875519 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875524 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875527 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875530 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:56:42.877103 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875534 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875537 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875540 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875542 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875545 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875548 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875551 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875553 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875556 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875558 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875561 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875563 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875569 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875572 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875575 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875578 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875580 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875583 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875586 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875589 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:56:42.877596 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875592 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875595 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875598 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875601 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875603 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875606 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875611 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875614 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875616 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875619 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875621 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875624 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875627 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875630 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875633 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875635 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875638 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875641 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875643 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875646 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:56:42.878118 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875649 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875652 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875654 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875658 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875664 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875667 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875672 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875675 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875678 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875695 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875699 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875702 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875705 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875708 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875711 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875714 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875716 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875719 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875721 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875724 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:56:42.878611 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875727 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875729 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875734 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875737 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875740 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875743 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875746 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875748 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875751 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875753 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875756 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875758 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875761 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875764 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875767 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875770 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.875774 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876187 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876192 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876195 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:56:42.879142 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876198 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876201 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876204 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876207 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876209 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876219 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876222 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876225 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876228 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876231 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876233 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876236 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876238 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876241 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876244 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876246 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876249 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876252 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876254 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876256 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:56:42.879644 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876259 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876262 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876265 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876267 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876270 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876272 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876275 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876278 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876282 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876287 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876290 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876294 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876297 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876299 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876302 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876304 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876307 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876310 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876321 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:56:42.880192 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876324 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876326 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876329 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876331 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876334 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876336 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876339 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876342 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876344 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876347 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876349 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876351 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876354 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876357 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876360 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876362 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876365 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876367 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876370 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:56:42.880709 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876374 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876378 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876382 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876385 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876388 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876391 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876393 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876396 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876399 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876401 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876404 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876407 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876409 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876420 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876423 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876426 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876428 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876431 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876435 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:56:42.881181 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876437 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876440 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876442 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876445 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876447 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876450 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876452 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876455 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876458 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876461 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876465 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876468 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876471 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876474 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876479 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876483 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876486 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876489 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.876492 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876582 2575 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876594 2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 17:56:42.881678 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876606 2575 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876613 2575 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876620 2575 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876623 2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876628 2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876633 2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876637 2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876640 2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876644 2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876647 2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876651 2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876654 2575 flags.go:64] FLAG: --cgroup-root="" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876657 2575 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876660 2575 flags.go:64] FLAG: --client-ca-file="" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876663 2575 flags.go:64] FLAG: --cloud-provider="external" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876666 2575 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876670 2575 flags.go:64] FLAG: --cluster-domain="" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876673 2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876676 2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876695 2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876699 2575 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876703 2575 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876707 2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 17:56:42.882224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876710 2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876714 2575 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876718 2575 flags.go:64] FLAG: --contention-profiling="false" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876721 2575 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876726 2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876729 2575 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876732 2575 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876736 2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876739 2575 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876742 2575 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876745 2575 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876749 2575 flags.go:64] FLAG: --enable-server="true" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876752 2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876757 2575 flags.go:64] FLAG: --event-burst="100" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876760 2575 flags.go:64] FLAG: --event-qps="50" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876763 2575 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876767 2575 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876770 2575 flags.go:64] FLAG: --eviction-hard="" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876774 2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876777 2575 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876780 2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876783 2575 flags.go:64] FLAG: --eviction-soft="" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876786 2575 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876788 2575 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876791 2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 17:56:42.882784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876794 2575 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876797 2575 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876800 2575 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876803 2575 flags.go:64] FLAG: --feature-gates="" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876806 2575 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876809 2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876813 2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876816 2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876819 2575 flags.go:64] FLAG: --healthz-port="10248" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876822 2575 flags.go:64] FLAG: --help="false" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876826 2575 flags.go:64] FLAG: --hostname-override="ip-10-0-135-147.ec2.internal" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876829 2575 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876832 2575 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876835 2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876838 2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876841 2575 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876844 2575 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876847 2575 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876850 2575 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876853 2575 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876856 2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876860 2575 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876862 2575 flags.go:64] FLAG: --kube-reserved="" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876865 2575 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 17:56:42.883408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876869 2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876873 2575 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876875 2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876878 2575 flags.go:64] FLAG: --lock-file="" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876881 2575 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876884 2575 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876887 2575 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876893 2575 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876896 2575 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876899 2575 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876902 2575 flags.go:64] FLAG: --logging-format="text" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876905 2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876908 2575 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876911 2575 flags.go:64] FLAG: --manifest-url="" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876913 2575 flags.go:64] FLAG: --manifest-url-header="" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876918 2575 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876921 2575 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876925 2575 flags.go:64] FLAG: --max-pods="110" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876929 2575 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876932 2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876935 2575 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876938 2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876941 2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876945 2575 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876948 2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 17:56:42.884097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876956 2575 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876959 2575 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876962 2575 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876965 2575 flags.go:64] FLAG: --pod-cidr="" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876968 2575 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876975 2575 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876979 2575 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876982 2575 flags.go:64] FLAG: --pods-per-core="0" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876993 2575 flags.go:64] FLAG: --port="10250" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876996 2575 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.876999 2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0fd84814e6bdd3bf2" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877003 2575 flags.go:64] FLAG: --qos-reserved="" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877006 2575 flags.go:64] FLAG: --read-only-port="10255" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877008 2575 flags.go:64] FLAG: --register-node="true" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877011 2575 flags.go:64] FLAG: --register-with-taints="" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877015 2575 flags.go:64] FLAG: --registry-burst="10" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877018 2575 flags.go:64] FLAG: --registry-qps="5" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877021 2575 flags.go:64] FLAG: --reserved-cpus="" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877024 2575 flags.go:64] FLAG: --reserved-memory="" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877030 2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877033 2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877036 2575 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877039 2575 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877042 2575 flags.go:64] FLAG: --runonce="false" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877045 2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 17:56:42.884737 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877048 2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877051 2575 flags.go:64] FLAG: --seccomp-default="false" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877053 2575 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877056 2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877059 2575 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877062 2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877065 2575 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877068 2575 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877071 2575 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877074 2575 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877077 2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877079 2575 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877082 2575 flags.go:64] FLAG: --system-cgroups="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877085 2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877091 2575 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877093 2575 flags.go:64] FLAG: --tls-cert-file="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877106 2575 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877111 2575 flags.go:64] FLAG: --tls-min-version="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877114 2575 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877117 2575 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877120 2575 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877123 2575 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877126 2575 flags.go:64] FLAG: --v="2" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877130 2575 flags.go:64] FLAG: --version="false" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877135 2575 flags.go:64] FLAG: --vmodule="" Jul 28 17:56:42.885380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877139 2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.877142 2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877296 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877300 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877304 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877307 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877312 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877315 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877320 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877322 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877325 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877328 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877330 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877333 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877336 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877338 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877342 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877346 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877350 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877353 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:56:42.886023 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877355 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877358 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877361 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877364 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877367 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877370 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877373 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877375 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877378 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877380 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877383 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877385 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877388 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877391 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877393 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877396 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877399 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877401 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877407 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:56:42.886526 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877409 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877413 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877416 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877419 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877421 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877425 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877429 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877432 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877435 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877438 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877441 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877444 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877448 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877451 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877454 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877457 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877460 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877463 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877468 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:56:42.887014 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877470 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877473 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877476 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877478 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877481 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877483 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877486 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877488 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877491 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877494 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877497 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877499 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877503 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877506 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877510 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877512 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877515 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877518 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877520 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877523 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:56:42.887483 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877525 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877528 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877530 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877533 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877535 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877538 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877541 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877543 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877546 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877548 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877551 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877553 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877557 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877559 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877562 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877564 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877567 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877569 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877572 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877574 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:56:42.888003 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877576 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877579 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.877582 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.878195 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.885421 2575 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.885440 2575 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.886074 2575 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.886083 2575 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886158 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886163 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886168 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886172 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886175 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886179 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:56:42.888490 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886182 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886185 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886188 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886191 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886194 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886198 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886202 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886204 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886208 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886211 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886215 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886218 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886221 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886225 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886227 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886230 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886233 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886235 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886238 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886240 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:56:42.888866 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886243 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886245 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886248 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886251 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886254 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886257 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886260 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886262 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886266 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886268 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886271 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886274 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886276 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886279 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886281 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886284 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886286 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886289 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886291 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886294 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:56:42.889360 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886296 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886299 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886302 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886305 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886309 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886312 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886315 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886317 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886320 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886322 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886325 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886327 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886330 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886332 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886335 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886338 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886342 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886344 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886347 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:56:42.889879 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886350 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886352 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886355 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886358 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886360 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886363 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886366 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886368 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886371 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886373 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886376 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886378 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886380 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886383 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886386 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886389 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886391 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886395 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886397 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:56:42.890359 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886400 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886402 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886405 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886407 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886410 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886412 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886415 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886417 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886420 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886424 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886427 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886429 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886432 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886434 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886437 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:56:42.890858 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.886442 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886567 2575 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886572 2575 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886575 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886579 2575 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886582 2575 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886584 2575 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886587 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886589 2575 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886592 2575 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886596 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886598 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886601 2575 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886603 2575 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886606 2575 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886608 2575 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886611 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886613 2575 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886616 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886618 2575 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 17:56:42.891232 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886621 2575 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886624 2575 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886626 2575 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886629 2575 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886631 2575 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886634 2575 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886637 2575 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886640 2575 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886643 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886645 2575 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886648 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886650 2575 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886653 2575 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886656 2575 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886658 2575 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886661 2575 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886664 2575 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886666 2575 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886668 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 17:56:42.891739 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886672 2575 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886675 2575 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886678 2575 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886697 2575 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886701 2575 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886705 2575 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886708 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886712 2575 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886716 2575 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886719 2575 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886722 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886725 2575 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886728 2575 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886731 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886734 2575 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886736 2575 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886739 2575 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886741 2575 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886744 2575 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 17:56:42.892210 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886747 2575 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886750 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886752 2575 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886755 2575 feature_gate.go:328] unrecognized feature gate: Example Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886758 2575 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886760 2575 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886763 2575 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886765 2575 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886768 2575 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886771 2575 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886773 2575 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886776 2575 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886779 2575 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886781 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886784 2575 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886786 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886789 2575 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886792 2575 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886794 2575 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886798 2575 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 17:56:42.892668 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886801 2575 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886804 2575 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886807 2575 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886810 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886813 2575 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886816 2575 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886819 2575 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886822 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886824 2575 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886827 2575 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886830 2575 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886832 2575 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886835 2575 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886838 2575 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886841 2575 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886843 2575 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886846 2575 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886849 2575 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886851 2575 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886854 2575 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 17:56:42.893265 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886856 2575 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:42.886859 2575 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.886864 2575 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.887001 2575 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.890969 2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.892367 2575 server.go:1019] "Starting client certificate rotation" Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.892473 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:56:42.893976 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.892515 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 17:56:42.897072 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.897044 2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:56:42.898357 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.898332 2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 17:56:42.911157 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.911113 2575 log.go:25] "Validated CRI v1 runtime API" Jul 28 17:56:42.914112 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.914083 2575 log.go:25] "Validated CRI v1 image API" Jul 28 17:56:42.915048 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.915026 2575 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 17:56:42.917597 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.917574 2575 fs.go:135] Filesystem UUIDs: map[5dc14b63-5484-4d82-ac30-4f16191dc16a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f3a29ff4-78f0-42f6-afbe-690d66e27579:/dev/nvme0n1p3] Jul 28 17:56:42.917663 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.917595 2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 17:56:42.922307 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.922159 2575 manager.go:216] Machine: {Timestamp:2026-07-28 17:56:42.921603189 +0000 UTC m=+0.336260182 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099995 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec278c8c4c0fcf2a339d32d6ff3d925e SystemUUID:ec278c8c-4c0f-cf2a-339d-32d6ff3d925e BootID:a48c644f-c329-4d4d-b386-06d26e079f26 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5f:a1:27:47:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5f:a1:27:47:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:ea:c1:b7:b7:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 17:56:42.922307 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.922301 2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 17:56:42.922458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.922433 2575 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 17:56:42.925636 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.925606 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:56:42.925769 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.925735 2575 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 17:56:42.925962 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.925767 2575 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-147.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 17:56:42.926464 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.926447 2575 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 17:56:42.926464 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.926462 2575 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 17:56:42.926591 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.926470 2575 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:56:42.927155 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.927140 2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 17:56:42.928477 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.928463 2575 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 17:56:42.932193 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932177 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:56:42.932385 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932374 2575 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 17:56:42.932417 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932407 2575 kubelet.go:486] "Attempting to sync node with API server" Jul 28 17:56:42.932455 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932422 2575 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 17:56:42.932455 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932437 2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 17:56:42.932455 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932448 2575 kubelet.go:398] "Adding apiserver pod source" Jul 28 17:56:42.932539 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932456 2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 17:56:42.932888 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932873 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:56:42.932946 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.932899 2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 17:56:42.935151 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.935134 2575 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 17:56:42.936988 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.936975 2575 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 17:56:42.937034 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.937006 2575 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 17:56:42.940109 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940077 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940124 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940134 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940144 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940157 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940167 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940177 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940183 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 17:56:42.940211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940211 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 17:56:42.940492 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940220 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 17:56:42.940492 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940254 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 17:56:42.940492 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.940341 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 17:56:42.941407 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.941385 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 17:56:42.941503 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.941413 2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 17:56:42.945603 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.945577 2575 server.go:1268] "Started kubelet" Jul 28 17:56:42.945897 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.945867 2575 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 17:56:42.946099 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.945855 2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 17:56:42.946184 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.946125 2575 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 17:56:42.946577 ip-10-0-135-147 systemd[1]: Started Kubernetes Kubelet. Jul 28 17:56:42.947325 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.947267 2575 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 17:56:42.948224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.948210 2575 server.go:310] "Adding debug handlers to kubelet server" Jul 28 17:56:42.948939 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.948915 2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-147.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 17:56:42.948939 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.948924 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 17:56:42.949089 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.949035 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 17:56:42.951439 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.951414 2575 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 17:56:42.954895 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.954875 2575 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 17:56:42.954979 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.954878 2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 17:56:42.955046 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.955012 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:42.955571 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955550 2575 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 17:56:42.955655 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955583 2575 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 17:56:42.955655 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955549 2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 17:56:42.955797 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955714 2575 reconstruct.go:97] "Volume reconstruction finished" Jul 28 17:56:42.955797 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955723 2575 reconciler.go:29] "Reconciler: start to sync state" Jul 28 17:56:42.955964 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955946 2575 factory.go:153] Registering CRI-O factory Jul 28 17:56:42.956015 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.955993 2575 factory.go:223] Registration of the crio container factory successfully Jul 28 17:56:42.956080 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.956067 2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 17:56:42.956080 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.956079 2575 factory.go:55] Registering systemd factory Jul 28 17:56:42.956163 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.956086 2575 factory.go:223] Registration of the systemd container factory successfully Jul 28 17:56:42.956163 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.956109 2575 factory.go:103] Registering Raw factory Jul 28 17:56:42.956163 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.956124 2575 manager.go:1196] Started watching for new ooms in manager Jul 28 17:56:42.956907 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.956890 2575 manager.go:349] Starting recovery of all containers Jul 28 17:56:42.957474 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.955934 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-147.ec2.internal.18c685c989c8b6f4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-147.ec2.internal,UID:ip-10-0-135-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-147.ec2.internal,},FirstTimestamp:2026-07-28 17:56:42.945533684 +0000 UTC m=+0.360190677,LastTimestamp:2026-07-28 17:56:42.945533684 +0000 UTC m=+0.360190677,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-147.ec2.internal,}" Jul 28 17:56:42.959893 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.959863 2575 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 17:56:42.966449 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.966408 2575 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-135-147.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 17:56:42.970755 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.970552 2575 manager.go:354] Recovery completed Jul 28 17:56:42.975357 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.975341 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:42.977635 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.977604 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:42.977635 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.977642 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:42.977812 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.977654 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:42.978254 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.978238 2575 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 17:56:42.978254 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.978251 2575 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 17:56:42.978386 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.978257 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:42.978386 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.978268 2575 state_mem.go:36] "Initialized new in-memory state store" Jul 28 17:56:42.979012 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.978993 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:42.979076 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.979027 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:42.979076 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.979039 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:42.979975 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.979900 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-147.ec2.internal.18c685c98bb2721d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-147.ec2.internal,UID:ip-10-0-135-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-147.ec2.internal,},FirstTimestamp:2026-07-28 17:56:42.977628701 +0000 UTC m=+0.392285697,LastTimestamp:2026-07-28 17:56:42.977628701 +0000 UTC m=+0.392285697,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-147.ec2.internal,}" Jul 28 17:56:42.980962 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.980950 2575 policy_none.go:49] "None policy: Start" Jul 28 17:56:42.981012 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.980968 2575 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 17:56:42.981012 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.980979 2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 17:56:42.982886 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:42.982871 2575 policy_none.go:47] "Start" Jul 28 17:56:42.991248 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:42.991161 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-147.ec2.internal.18c685c98bb2bebd default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-147.ec2.internal,UID:ip-10-0-135-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-135-147.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-135-147.ec2.internal,},FirstTimestamp:2026-07-28 17:56:42.977648317 +0000 UTC m=+0.392305310,LastTimestamp:2026-07-28 17:56:42.977648317 +0000 UTC m=+0.392305310,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-147.ec2.internal,}" Jul 28 17:56:43.003644 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.003534 2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-147.ec2.internal.18c685c98bb2e73f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-147.ec2.internal,UID:ip-10-0-135-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-135-147.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-135-147.ec2.internal,},FirstTimestamp:2026-07-28 17:56:42.977658687 +0000 UTC m=+0.392315680,LastTimestamp:2026-07-28 17:56:42.977658687 +0000 UTC m=+0.392315680,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-147.ec2.internal,}" Jul 28 17:56:43.006017 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.005991 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjcpg" Jul 28 17:56:43.014138 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.014112 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sjcpg" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.014231 2575 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-135-147.ec2.internal.18c685c98bb2721d\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-147.ec2.internal.18c685c98bb2721d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-147.ec2.internal,UID:ip-10-0-135-147.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-135-147.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-135-147.ec2.internal,},FirstTimestamp:2026-07-28 17:56:42.977628701 +0000 UTC m=+0.392285697,LastTimestamp:2026-07-28 17:56:42.979012844 +0000 UTC m=+0.393669842,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-147.ec2.internal,}" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.019411 2575 manager.go:337] "Starting Device Plugin manager" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.019455 2575 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.019466 2575 server.go:87] "Starting device plugin registration server" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.019801 2575 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.019815 2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.019992 2575 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.020054 2575 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.020059 2575 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.020740 2575 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 17:56:43.035419 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.020785 2575 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.051320 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.051284 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 17:56:43.052542 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.052524 2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 17:56:43.052633 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.052569 2575 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 17:56:43.052633 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.052595 2575 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 17:56:43.052760 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.052638 2575 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 17:56:43.055619 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.055598 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:56:43.120049 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.119983 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:43.121011 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.120991 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:43.121114 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.121028 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:43.121114 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.121038 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:43.121114 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.121064 2575 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.133365 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.133341 2575 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.133365 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.133370 2575 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-147.ec2.internal\": node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.153164 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.153130 2575 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal"] Jul 28 17:56:43.153330 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.153213 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:43.154728 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.154709 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:43.154849 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.154739 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:43.154849 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.154748 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:43.157102 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.157078 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:43.157207 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.157188 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.157245 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.157224 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:43.157985 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.157968 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:43.158108 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.157995 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:43.158108 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.158005 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:43.158108 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.157969 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:43.158108 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.158079 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:43.158108 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.158089 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:43.160296 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.160278 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.160412 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.160319 2575 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 17:56:43.161200 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.161183 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientMemory" Jul 28 17:56:43.161304 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.161214 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 17:56:43.161304 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.161225 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeHasSufficientPID" Jul 28 17:56:43.183335 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.183306 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.191769 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.191747 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-147.ec2.internal\" not found" node="ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.196376 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.196355 2575 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-147.ec2.internal\" not found" node="ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.256927 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.256891 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/545c1a00c368df77d9e72312b6e155db-config\") pod \"kube-apiserver-proxy-ip-10-0-135-147.ec2.internal\" (UID: \"545c1a00c368df77d9e72312b6e155db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.256927 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.256927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5071a057a738bd7dfffd7bae34454265-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal\" (UID: \"5071a057a738bd7dfffd7bae34454265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.257095 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.256947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5071a057a738bd7dfffd7bae34454265-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal\" (UID: \"5071a057a738bd7dfffd7bae34454265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.283876 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.283843 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.357784 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.357757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5071a057a738bd7dfffd7bae34454265-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal\" (UID: \"5071a057a738bd7dfffd7bae34454265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.357912 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.357789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5071a057a738bd7dfffd7bae34454265-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal\" (UID: \"5071a057a738bd7dfffd7bae34454265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.357912 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.357811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/545c1a00c368df77d9e72312b6e155db-config\") pod \"kube-apiserver-proxy-ip-10-0-135-147.ec2.internal\" (UID: \"545c1a00c368df77d9e72312b6e155db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.357912 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.357845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/545c1a00c368df77d9e72312b6e155db-config\") pod \"kube-apiserver-proxy-ip-10-0-135-147.ec2.internal\" (UID: \"545c1a00c368df77d9e72312b6e155db\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.357912 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.357853 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5071a057a738bd7dfffd7bae34454265-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal\" (UID: \"5071a057a738bd7dfffd7bae34454265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.357912 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.357869 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5071a057a738bd7dfffd7bae34454265-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal\" (UID: \"5071a057a738bd7dfffd7bae34454265\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.384900 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.384826 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.486195 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.486144 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.493884 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.493854 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.500458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.500435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.588552 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.588509 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.690588 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.690524 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.784849 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.784816 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:56:43.791201 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.791170 2575 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-147.ec2.internal\" not found" Jul 28 17:56:43.852434 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.852398 2575 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:56:43.855946 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.855920 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.874282 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.874248 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:56:43.876530 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.876512 2575 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" Jul 28 17:56:43.889971 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.889943 2575 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 17:56:43.892309 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.892278 2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 17:56:43.892499 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.892464 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:56:43.892499 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.892464 2575 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 17:56:43.934066 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.934027 2575 apiserver.go:52] "Watching apiserver" Jul 28 17:56:43.944596 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.944497 2575 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 17:56:43.944959 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.944933 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qrp7l","openshift-network-diagnostics/network-check-target-2qsvk","kube-system/konnectivity-agent-6gtgh","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw","openshift-cluster-node-tuning-operator/tuned-wcfrz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal","openshift-multus/multus-b44c7","openshift-multus/network-metrics-daemon-fjvtt","openshift-network-operator/iptables-alerter-sg44n","openshift-ovn-kubernetes/ovnkube-node-2ck7j","kube-system/global-pull-secret-syncer-ztp68","kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal","openshift-dns/node-resolver-mghpz","openshift-image-registry/node-ca-rgqsk"] Jul 28 17:56:43.947782 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.947755 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.950104 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.950080 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.951754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.951734 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.951870 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.951854 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 17:56:43.952279 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.952264 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:43.952345 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.952325 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:43.952494 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.952476 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 17:56:43.952982 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.952965 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.953029 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.953007 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fd2r7\"" Jul 28 17:56:43.954594 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.954579 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.955041 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.955023 2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 17:56:43.957042 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.957025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:43.957105 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.957091 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:43.959441 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.959423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:43.961556 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961529 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.961662 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961564 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.961662 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-netns\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.961662 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961605 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-kubelet\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.961662 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961654 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-hostroot\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961668 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961680 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vtjv\" (UniqueName: \"kubernetes.io/projected/144d01ab-98a5-4f4c-bee9-a4030a422bec-kube-api-access-9vtjv\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961708 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961731 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-device-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961758 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961804 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkfcj\" (UniqueName: \"kubernetes.io/projected/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-kube-api-access-hkfcj\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961828 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-cni-bin\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961855 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-cni-multus\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-multus-certs\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cnibin\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961943 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.961997 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c88eb666-c963-4a96-a1cf-67097790c4df-multus-daemon-config\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962034 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zgf\" (UniqueName: \"kubernetes.io/projected/c88eb666-c963-4a96-a1cf-67097790c4df-kube-api-access-j8zgf\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-system-cni-dir\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962345 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962114 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkjbp\" (UniqueName: \"kubernetes.io/projected/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-kube-api-access-bkjbp\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962164 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-system-cni-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962194 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-socket-dir-parent\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962233 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-cni-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962255 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-cnibin\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-os-release\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962303 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c88eb666-c963-4a96-a1cf-67097790c4df-cni-binary-copy\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962324 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-os-release\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962382 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962427 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-conf-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-etc-kubernetes\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962476 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:43.962854 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.962518 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-k8s-cni-cncf-io\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:43.964151 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.964130 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:43.966672 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.966649 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:43.966809 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.966751 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.966932 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.966913 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mgxst\"" Jul 28 17:56:43.967128 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967111 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.967380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967359 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rdxdr\"" Jul 28 17:56:43.967380 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967367 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-fd8pz\"" Jul 28 17:56:43.967497 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967413 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.967622 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967605 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.967664 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjqx7\"" Jul 28 17:56:43.967664 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967653 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 17:56:43.967961 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967921 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 17:56:43.968078 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967977 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.968078 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967987 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.968078 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.968024 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 17:56:43.968078 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.968060 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.968078 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967977 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 17:56:43.968389 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.967983 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qpx4p\"" Jul 28 17:56:43.968389 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.968307 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.969392 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.969375 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:43.969456 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:43.969436 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:43.971015 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.970755 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 17:56:43.971015 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.970891 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 17:56:43.972028 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.971807 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 17:56:43.972028 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.971978 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.973583 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.972300 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.973583 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.972300 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-xtwq9\"" Jul 28 17:56:43.974215 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.974167 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 17:56:43.974530 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.974512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:43.977773 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.977748 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:43.982580 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982552 2575 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 17:56:43.982720 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982585 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 17:56:43.982720 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982557 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 17:56:43.982720 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982611 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 17:56:43.982720 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982656 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 17:56:43.982720 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982680 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 17:56:43.982967 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.982953 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-99zv4\"" Jul 28 17:56:43.983078 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:43.983062 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qcstf\"" Jul 28 17:56:44.020286 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.020207 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:51:43 +0000 UTC" deadline="2028-05-07 05:58:19.271820212 +0000 UTC" Jul 28 17:56:44.020286 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.020280 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15564h1m35.25154257s" Jul 28 17:56:44.020286 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.020284 2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjd9g" Jul 28 17:56:44.035526 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.035495 2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xjd9g" Jul 28 17:56:44.057135 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.057113 2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 17:56:44.063404 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063377 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-multus-certs\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cnibin\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysctl-conf\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063449 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-slash\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063467 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063490 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-ovnkube-config\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cnibin\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-multus-certs\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zgf\" (UniqueName: \"kubernetes.io/projected/c88eb666-c963-4a96-a1cf-67097790c4df-kube-api-access-j8zgf\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063572 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-system-cni-dir\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063587 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj465\" (UniqueName: \"kubernetes.io/projected/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-kube-api-access-sj465\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063604 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-systemd\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063621 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-system-cni-dir\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063642 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61ce049e-c323-430a-95c0-fa3618049ab1-ovn-node-metrics-cert\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063678 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df10c201-bf85-4f6c-b7b1-e79bc719405c-konnectivity-ca\") pod \"konnectivity-agent-6gtgh\" (UID: \"df10c201-bf85-4f6c-b7b1-e79bc719405c\") " pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063707 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-run\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-socket-dir-parent\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063756 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-cni-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063792 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c88eb666-c963-4a96-a1cf-67097790c4df-cni-binary-copy\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063816 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-os-release\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063838 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98740ddb-0e74-4f8a-9e02-615ea70bc398-serviceca\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063793 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-socket-dir-parent\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.063885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-os-release\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063923 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063942 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-etc-kubernetes\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063950 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-cni-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysconfig\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063984 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clwxh\" (UniqueName: \"kubernetes.io/projected/61ce049e-c323-430a-95c0-fa3618049ab1-kube-api-access-clwxh\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.063985 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-etc-kubernetes\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-hosts-file\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064039 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-tmp-dir\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysctl-d\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064070 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-host\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-var-lib-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064145 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-etc-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064159 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-node-log\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064186 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4s79\" (UniqueName: \"kubernetes.io/projected/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-kube-api-access-s4s79\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.064481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064219 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-host-slash\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064234 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-var-lib-kubelet\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064256 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-ovnkube-script-lib\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064280 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63e15748-3635-4ac4-be53-09c308ba90bb-kubelet-config\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-netns\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064329 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-kubelet\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064349 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vtjv\" (UniqueName: \"kubernetes.io/projected/144d01ab-98a5-4f4c-bee9-a4030a422bec-kube-api-access-9vtjv\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064360 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-netns\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064381 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-device-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064415 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-device-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064430 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c88eb666-c963-4a96-a1cf-67097790c4df-cni-binary-copy\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064502 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cfc68e8-0838-4804-9704-d95d306e26df-tmp\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064500 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-kubelet\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064532 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cfc68e8-0838-4804-9704-d95d306e26df-etc-tuned\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064595 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkfcj\" (UniqueName: \"kubernetes.io/projected/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-kube-api-access-hkfcj\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064619 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.065292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-cni-bin\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064649 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-cni-multus\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064663 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-cni-bin\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064668 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064718 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-ovn\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064759 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-var-lib-cni-multus\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064751 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjpdh\" (UniqueName: \"kubernetes.io/projected/98740ddb-0e74-4f8a-9e02-615ea70bc398-kube-api-access-rjpdh\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c88eb666-c963-4a96-a1cf-67097790c4df-multus-daemon-config\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064835 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064842 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkjbp\" (UniqueName: \"kubernetes.io/projected/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-kube-api-access-bkjbp\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-sys\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064926 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-lib-modules\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.064948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-systemd-units\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065026 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-systemd\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065058 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:44.066000 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-system-cni-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-kubelet\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-cni-netd\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065152 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-system-cni-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065185 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065209 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98740ddb-0e74-4f8a-9e02-615ea70bc398-host\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065242 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-cnibin\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065270 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-os-release\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065317 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-cnibin\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065325 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065355 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065362 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-os-release\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065369 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cni-binary-copy\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065386 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-conf-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.065437 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065442 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-iptables-alerter-script\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065470 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-modprobe-d\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.066561 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.065503 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:56:44.565471785 +0000 UTC m=+1.980128767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c88eb666-c963-4a96-a1cf-67097790c4df-multus-daemon-config\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-kubernetes\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065509 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-multus-conf-dir\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065562 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-env-overrides\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065592 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065619 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-k8s-cni-cncf-io\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065644 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63e15748-3635-4ac4-be53-09c308ba90bb-dbus\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df10c201-bf85-4f6c-b7b1-e79bc719405c-agent-certs\") pod \"konnectivity-agent-6gtgh\" (UID: \"df10c201-bf85-4f6c-b7b1-e79bc719405c\") " pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065705 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065727 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065800 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065705 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-host-run-k8s-cni-cncf-io\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-hostroot\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.067181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065901 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcqp8\" (UniqueName: \"kubernetes.io/projected/0cfc68e8-0838-4804-9704-d95d306e26df-kube-api-access-fcqp8\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.067754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065922 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-run-netns\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.067754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065922 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c88eb666-c963-4a96-a1cf-67097790c4df-hostroot\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.067754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065938 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-log-socket\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.067754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065968 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-cni-bin\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.067754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.065969 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.084397 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.083925 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:56:44.084397 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.083962 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:56:44.084397 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.083995 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:44.084397 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.084048 2575 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 17:56:44.084397 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.084359 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:56:44.58433332 +0000 UTC m=+1.998990302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:44.087886 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.087857 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkfcj\" (UniqueName: \"kubernetes.io/projected/dc3dfca2-b36c-4613-b67f-1b2aaf7567d9-kube-api-access-hkfcj\") pod \"aws-ebs-csi-driver-node-7rvpw\" (UID: \"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.087886 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.087870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vtjv\" (UniqueName: \"kubernetes.io/projected/144d01ab-98a5-4f4c-bee9-a4030a422bec-kube-api-access-9vtjv\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:44.088200 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.088176 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkjbp\" (UniqueName: \"kubernetes.io/projected/9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae-kube-api-access-bkjbp\") pod \"multus-additional-cni-plugins-qrp7l\" (UID: \"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae\") " pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.088348 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.088333 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zgf\" (UniqueName: \"kubernetes.io/projected/c88eb666-c963-4a96-a1cf-67097790c4df-kube-api-access-j8zgf\") pod \"multus-b44c7\" (UID: \"c88eb666-c963-4a96-a1cf-67097790c4df\") " pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.167377 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167338 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.167377 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167379 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61ce049e-c323-430a-95c0-fa3618049ab1-ovn-node-metrics-cert\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.167618 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167402 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df10c201-bf85-4f6c-b7b1-e79bc719405c-konnectivity-ca\") pod \"konnectivity-agent-6gtgh\" (UID: \"df10c201-bf85-4f6c-b7b1-e79bc719405c\") " pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.167618 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167418 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-run\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.167618 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98740ddb-0e74-4f8a-9e02-615ea70bc398-serviceca\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.167618 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167467 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.167618 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167511 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysconfig\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.167618 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-run\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.168042 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysconfig\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.168171 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.167544 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clwxh\" (UniqueName: \"kubernetes.io/projected/61ce049e-c323-430a-95c0-fa3618049ab1-kube-api-access-clwxh\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.168283 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-hosts-file\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.168354 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-hosts-file\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.168413 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168393 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-tmp-dir\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.168469 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168431 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysctl-d\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.168592 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168550 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-host\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.168655 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-var-lib-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.168655 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168645 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysctl-d\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.168785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168664 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/df10c201-bf85-4f6c-b7b1-e79bc719405c-konnectivity-ca\") pod \"konnectivity-agent-6gtgh\" (UID: \"df10c201-bf85-4f6c-b7b1-e79bc719405c\") " pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.168785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/98740ddb-0e74-4f8a-9e02-615ea70bc398-serviceca\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.168785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168694 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-etc-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.168785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168734 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-host\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.168785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168761 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-node-log\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168785 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-var-lib-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-etc-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168805 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4s79\" (UniqueName: \"kubernetes.io/projected/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-kube-api-access-s4s79\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.169020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168833 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-node-log\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-host-slash\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.169020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.168975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-var-lib-kubelet\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.169321 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169064 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-ovnkube-script-lib\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169321 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63e15748-3635-4ac4-be53-09c308ba90bb-kubelet-config\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.169321 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169142 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-host-slash\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.169321 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169200 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cfc68e8-0838-4804-9704-d95d306e26df-tmp\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.169321 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cfc68e8-0838-4804-9704-d95d306e26df-etc-tuned\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.169587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169334 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-ovn\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjpdh\" (UniqueName: \"kubernetes.io/projected/98740ddb-0e74-4f8a-9e02-615ea70bc398-kube-api-access-rjpdh\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.169587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169428 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-sys\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.169587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169459 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-lib-modules\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.169587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169514 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-systemd-units\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169547 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-systemd\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169687 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-kubelet\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169726 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-cni-netd\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.169918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.169918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98740ddb-0e74-4f8a-9e02-615ea70bc398-host\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.169918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169878 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-iptables-alerter-script\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.170166 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169920 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-modprobe-d\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.170166 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.169975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-kubernetes\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.170166 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170043 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-env-overrides\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170166 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170108 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63e15748-3635-4ac4-be53-09c308ba90bb-dbus\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.170166 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-ovnkube-script-lib\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170479 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170168 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df10c201-bf85-4f6c-b7b1-e79bc719405c-agent-certs\") pod \"konnectivity-agent-6gtgh\" (UID: \"df10c201-bf85-4f6c-b7b1-e79bc719405c\") " pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.170479 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170212 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-systemd\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170479 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170479 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170320 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcqp8\" (UniqueName: \"kubernetes.io/projected/0cfc68e8-0838-4804-9704-d95d306e26df-kube-api-access-fcqp8\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.170479 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170360 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-run-netns\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170539 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-log-socket\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170551 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-tmp-dir\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.170722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170576 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-cni-bin\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170632 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysctl-conf\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.170722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170680 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-kubelet\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.170722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170710 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-slash\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.171051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170745 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.171051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170754 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-cni-netd\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.171051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170811 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-ovnkube-config\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.171051 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.170896 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:44.171051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/63e15748-3635-4ac4-be53-09c308ba90bb-kubelet-config\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.171291 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.171111 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:56:44.671089436 +0000 UTC m=+2.085746434 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:44.171900 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.171461 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/98740ddb-0e74-4f8a-9e02-615ea70bc398-host\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.171900 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.171538 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-run-ovn-kubernetes\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.172074 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.171892 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-run-netns\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.172074 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.171974 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-log-socket\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.172074 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172049 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj465\" (UniqueName: \"kubernetes.io/projected/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-kube-api-access-sj465\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.172235 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172151 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-systemd-units\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.172235 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172166 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-sys\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.172338 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172227 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-lib-modules\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.172338 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172302 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-kubernetes\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.172465 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-ovn\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.172611 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/63e15748-3635-4ac4-be53-09c308ba90bb-dbus\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.172710 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-systemd\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.172710 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.170636 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-cni-bin\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.173132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172851 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-run-openvswitch\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.173132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.172866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-systemd\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.173132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173073 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-sysctl-conf\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.173132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-etc-modprobe-d\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.174187 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0cfc68e8-0838-4804-9704-d95d306e26df-var-lib-kubelet\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.174187 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173243 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-env-overrides\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.174187 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/61ce049e-c323-430a-95c0-fa3618049ab1-host-slash\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.174187 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173512 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/61ce049e-c323-430a-95c0-fa3618049ab1-ovnkube-config\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.174187 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.173608 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-iptables-alerter-script\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.174187 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.174074 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/61ce049e-c323-430a-95c0-fa3618049ab1-ovn-node-metrics-cert\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.175113 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.175087 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/df10c201-bf85-4f6c-b7b1-e79bc719405c-agent-certs\") pod \"konnectivity-agent-6gtgh\" (UID: \"df10c201-bf85-4f6c-b7b1-e79bc719405c\") " pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.175113 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.175100 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0cfc68e8-0838-4804-9704-d95d306e26df-etc-tuned\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.175645 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.175628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0cfc68e8-0838-4804-9704-d95d306e26df-tmp\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.185596 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.185559 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clwxh\" (UniqueName: \"kubernetes.io/projected/61ce049e-c323-430a-95c0-fa3618049ab1-kube-api-access-clwxh\") pod \"ovnkube-node-2ck7j\" (UID: \"61ce049e-c323-430a-95c0-fa3618049ab1\") " pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.190517 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.190479 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj465\" (UniqueName: \"kubernetes.io/projected/0e18bfb5-2d47-4cfc-8f64-86b1a23606a1-kube-api-access-sj465\") pod \"iptables-alerter-sg44n\" (UID: \"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1\") " pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.190684 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.190525 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjpdh\" (UniqueName: \"kubernetes.io/projected/98740ddb-0e74-4f8a-9e02-615ea70bc398-kube-api-access-rjpdh\") pod \"node-ca-rgqsk\" (UID: \"98740ddb-0e74-4f8a-9e02-615ea70bc398\") " pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.190754 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.190687 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4s79\" (UniqueName: \"kubernetes.io/projected/a3ba8fda-d419-4c09-880f-8a768f7cd1a0-kube-api-access-s4s79\") pod \"node-resolver-mghpz\" (UID: \"a3ba8fda-d419-4c09-880f-8a768f7cd1a0\") " pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.190963 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.190944 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcqp8\" (UniqueName: \"kubernetes.io/projected/0cfc68e8-0838-4804-9704-d95d306e26df-kube-api-access-fcqp8\") pod \"tuned-wcfrz\" (UID: \"0cfc68e8-0838-4804-9704-d95d306e26df\") " pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.217972 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.217927 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5071a057a738bd7dfffd7bae34454265.slice/crio-70a7d2a4f56240769093df06ec8c57dae771dbca03857f79cf1e8d095b984345 WatchSource:0}: Error finding container 70a7d2a4f56240769093df06ec8c57dae771dbca03857f79cf1e8d095b984345: Status 404 returned error can't find the container with id 70a7d2a4f56240769093df06ec8c57dae771dbca03857f79cf1e8d095b984345 Jul 28 17:56:44.218193 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.218177 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod545c1a00c368df77d9e72312b6e155db.slice/crio-d0167e5a9673d59c02aa2d5d45f67445994c6f515e9284f4e407430ade8dda15 WatchSource:0}: Error finding container d0167e5a9673d59c02aa2d5d45f67445994c6f515e9284f4e407430ade8dda15: Status 404 returned error can't find the container with id d0167e5a9673d59c02aa2d5d45f67445994c6f515e9284f4e407430ade8dda15 Jul 28 17:56:44.224135 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.224114 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 17:56:44.275645 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.275600 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-b44c7" Jul 28 17:56:44.282582 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.282541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" Jul 28 17:56:44.286575 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.286547 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88eb666_c963_4a96_a1cf_67097790c4df.slice/crio-c3bdd4df4674bd591c798e8a7c0f8c56f63d699c2b8ed3c17d145e4faecb4aa6 WatchSource:0}: Error finding container c3bdd4df4674bd591c798e8a7c0f8c56f63d699c2b8ed3c17d145e4faecb4aa6: Status 404 returned error can't find the container with id c3bdd4df4674bd591c798e8a7c0f8c56f63d699c2b8ed3c17d145e4faecb4aa6 Jul 28 17:56:44.292979 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.292951 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c0b6a3f_dcd9_48bf_8cb0_9c42e2de7dae.slice/crio-20103d994d616a8e303ae785f4f8ef29fdbbdde660a461ca663392e2e95738c7 WatchSource:0}: Error finding container 20103d994d616a8e303ae785f4f8ef29fdbbdde660a461ca663392e2e95738c7: Status 404 returned error can't find the container with id 20103d994d616a8e303ae785f4f8ef29fdbbdde660a461ca663392e2e95738c7 Jul 28 17:56:44.303360 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.303334 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" Jul 28 17:56:44.311539 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.311509 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-mghpz" Jul 28 17:56:44.315297 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.315243 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc3dfca2_b36c_4613_b67f_1b2aaf7567d9.slice/crio-9e4d0edc06f54fa1ed844dcaca16161fc626b49de6a5e2f5b584528223933e3a WatchSource:0}: Error finding container 9e4d0edc06f54fa1ed844dcaca16161fc626b49de6a5e2f5b584528223933e3a: Status 404 returned error can't find the container with id 9e4d0edc06f54fa1ed844dcaca16161fc626b49de6a5e2f5b584528223933e3a Jul 28 17:56:44.320166 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.320138 2575 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:56:44.322426 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.322393 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ba8fda_d419_4c09_880f_8a768f7cd1a0.slice/crio-311e16263b05ba5ee1d0d1341c5b64c99af419cb94534b915037ff8fc9006676 WatchSource:0}: Error finding container 311e16263b05ba5ee1d0d1341c5b64c99af419cb94534b915037ff8fc9006676: Status 404 returned error can't find the container with id 311e16263b05ba5ee1d0d1341c5b64c99af419cb94534b915037ff8fc9006676 Jul 28 17:56:44.331886 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.331857 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-sg44n" Jul 28 17:56:44.338458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.338435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" Jul 28 17:56:44.342546 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.342522 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e18bfb5_2d47_4cfc_8f64_86b1a23606a1.slice/crio-b08229096919457f71d46ee68e0d21818eae466d1acc8b242fa495be8565f8d7 WatchSource:0}: Error finding container b08229096919457f71d46ee68e0d21818eae466d1acc8b242fa495be8565f8d7: Status 404 returned error can't find the container with id b08229096919457f71d46ee68e0d21818eae466d1acc8b242fa495be8565f8d7 Jul 28 17:56:44.348171 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.348145 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cfc68e8_0838_4804_9704_d95d306e26df.slice/crio-506252f45aeaccb593f71edb4370eabb2b041caa70ad96766b00a1f8586b45c9 WatchSource:0}: Error finding container 506252f45aeaccb593f71edb4370eabb2b041caa70ad96766b00a1f8586b45c9: Status 404 returned error can't find the container with id 506252f45aeaccb593f71edb4370eabb2b041caa70ad96766b00a1f8586b45c9 Jul 28 17:56:44.371183 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.371139 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:56:44.382576 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.382545 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ce049e_c323_430a_95c0_fa3618049ab1.slice/crio-d0bb6ecfecf8ced284c7cfd5bc06fb8b17465cf77afef1709f354e6a5451eace WatchSource:0}: Error finding container d0bb6ecfecf8ced284c7cfd5bc06fb8b17465cf77afef1709f354e6a5451eace: Status 404 returned error can't find the container with id d0bb6ecfecf8ced284c7cfd5bc06fb8b17465cf77afef1709f354e6a5451eace Jul 28 17:56:44.396808 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.396781 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:56:44.402973 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.402944 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rgqsk" Jul 28 17:56:44.407451 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.407424 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf10c201_bf85_4f6c_b7b1_e79bc719405c.slice/crio-513d4323ae477c5559055536032cb26217bd5f42cf82d820889f0b14d57910f8 WatchSource:0}: Error finding container 513d4323ae477c5559055536032cb26217bd5f42cf82d820889f0b14d57910f8: Status 404 returned error can't find the container with id 513d4323ae477c5559055536032cb26217bd5f42cf82d820889f0b14d57910f8 Jul 28 17:56:44.413390 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:56:44.413362 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98740ddb_0e74_4f8a_9e02_615ea70bc398.slice/crio-79594c62d2af5c8c2889053473011b64d6afd8e5541f099632793fa20c35b648 WatchSource:0}: Error finding container 79594c62d2af5c8c2889053473011b64d6afd8e5541f099632793fa20c35b648: Status 404 returned error can't find the container with id 79594c62d2af5c8c2889053473011b64d6afd8e5541f099632793fa20c35b648 Jul 28 17:56:44.415609 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.415587 2575 kuberuntime_manager.go:1459] "Unhandled Error" err="container node-ca start failed in pod node-ca-rgqsk_openshift-image-registry(98740ddb-0e74-4f8a-9e02-615ea70bc398): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 28 17:56:44.415674 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.415623 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ErrImagePull: \"pull QPS exceeded\"" pod="openshift-image-registry/node-ca-rgqsk" podUID="98740ddb-0e74-4f8a-9e02-615ea70bc398" Jul 28 17:56:44.576553 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.576452 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:44.576733 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.576600 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:44.576733 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.576660 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:56:45.576644924 +0000 UTC m=+2.991301906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:44.677563 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.677493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:44.677563 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:44.677566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:44.677856 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.677727 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:56:44.677856 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.677758 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:56:44.677856 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.677779 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:44.677856 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.677725 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:44.677856 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.677840 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:56:45.677817336 +0000 UTC m=+3.092474318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:44.678191 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:44.677868 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:56:45.677855723 +0000 UTC m=+3.092512714 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:45.039713 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.039546 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:51:44 +0000 UTC" deadline="2028-03-06 04:58:38.596645842 +0000 UTC" Jul 28 17:56:45.039713 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.039634 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14075h1m53.557017714s" Jul 28 17:56:45.067972 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.067905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6gtgh" event={"ID":"df10c201-bf85-4f6c-b7b1-e79bc719405c","Type":"ContainerStarted","Data":"513d4323ae477c5559055536032cb26217bd5f42cf82d820889f0b14d57910f8"} Jul 28 17:56:45.084307 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.084261 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"d0bb6ecfecf8ced284c7cfd5bc06fb8b17465cf77afef1709f354e6a5451eace"} Jul 28 17:56:45.110197 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.110146 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" event={"ID":"0cfc68e8-0838-4804-9704-d95d306e26df","Type":"ContainerStarted","Data":"506252f45aeaccb593f71edb4370eabb2b041caa70ad96766b00a1f8586b45c9"} Jul 28 17:56:45.125065 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.124977 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sg44n" event={"ID":"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1","Type":"ContainerStarted","Data":"b08229096919457f71d46ee68e0d21818eae466d1acc8b242fa495be8565f8d7"} Jul 28 17:56:45.136371 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.136325 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" event={"ID":"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9","Type":"ContainerStarted","Data":"9e4d0edc06f54fa1ed844dcaca16161fc626b49de6a5e2f5b584528223933e3a"} Jul 28 17:56:45.138402 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.138374 2575 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:56:45.148382 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.148334 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerStarted","Data":"20103d994d616a8e303ae785f4f8ef29fdbbdde660a461ca663392e2e95738c7"} Jul 28 17:56:45.166802 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.166743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b44c7" event={"ID":"c88eb666-c963-4a96-a1cf-67097790c4df","Type":"ContainerStarted","Data":"c3bdd4df4674bd591c798e8a7c0f8c56f63d699c2b8ed3c17d145e4faecb4aa6"} Jul 28 17:56:45.174649 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.174589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" event={"ID":"545c1a00c368df77d9e72312b6e155db","Type":"ContainerStarted","Data":"d0167e5a9673d59c02aa2d5d45f67445994c6f515e9284f4e407430ade8dda15"} Jul 28 17:56:45.185647 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.184743 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rgqsk" event={"ID":"98740ddb-0e74-4f8a-9e02-615ea70bc398","Type":"ContainerStarted","Data":"79594c62d2af5c8c2889053473011b64d6afd8e5541f099632793fa20c35b648"} Jul 28 17:56:45.188287 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.187961 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"node-ca\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:843b51893f8a3884f1fe56948902e4dd166193b21cc34e099aa8d96a8ff2ea09\\\": ErrImagePull: pull QPS exceeded\"" pod="openshift-image-registry/node-ca-rgqsk" podUID="98740ddb-0e74-4f8a-9e02-615ea70bc398" Jul 28 17:56:45.194839 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.194735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mghpz" event={"ID":"a3ba8fda-d419-4c09-880f-8a768f7cd1a0","Type":"ContainerStarted","Data":"311e16263b05ba5ee1d0d1341c5b64c99af419cb94534b915037ff8fc9006676"} Jul 28 17:56:45.202608 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.202566 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" event={"ID":"5071a057a738bd7dfffd7bae34454265","Type":"ContainerStarted","Data":"70a7d2a4f56240769093df06ec8c57dae771dbca03857f79cf1e8d095b984345"} Jul 28 17:56:45.580523 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.580487 2575 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 17:56:45.586334 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.586297 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:45.586524 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.586485 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:45.586589 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.586555 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:56:47.586534078 +0000 UTC m=+5.001191082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:45.687607 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.687561 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:45.687836 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:45.687627 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:45.687836 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.687753 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:56:45.687836 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.687775 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:45.687836 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.687786 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:56:45.687836 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.687804 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:45.688125 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.687845 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:56:47.687826116 +0000 UTC m=+5.102483119 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:45.688125 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:45.687865 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:56:47.687855962 +0000 UTC m=+5.102512945 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:46.040114 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:46.040012 2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:51:44 +0000 UTC" deadline="2028-04-01 20:33:05.412700081 +0000 UTC" Jul 28 17:56:46.040114 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:46.040061 2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14714h36m19.372645743s" Jul 28 17:56:46.053527 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:46.053489 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:46.053701 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:46.053633 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:46.053785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:46.053739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:46.053845 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:46.053822 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:46.053924 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:46.053910 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:46.054033 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:46.054004 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:47.606189 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:47.606153 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:47.606730 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.606318 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:47.606730 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.606384 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:56:51.606364714 +0000 UTC m=+9.021021695 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:47.706896 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:47.706852 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:47.707100 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:47.706915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:47.707100 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.707063 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:47.707218 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.707128 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:56:51.70711288 +0000 UTC m=+9.121769861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:47.707795 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.707565 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:56:47.707795 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.707588 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:56:47.707795 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.707603 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:47.707795 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:47.707656 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:56:51.707637445 +0000 UTC m=+9.122294433 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:48.054645 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:48.053768 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:48.054645 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:48.053921 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:48.054645 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:48.054326 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:48.054645 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:48.054432 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:48.054645 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:48.054488 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:48.054645 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:48.054578 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:50.053271 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:50.053227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:50.053271 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:50.053261 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:50.053861 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:50.053228 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:50.053861 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:50.053384 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:50.053861 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:50.053507 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:50.053861 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:50.053601 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:51.642891 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:51.642845 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:51.643403 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.643001 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:51.643403 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.643089 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:56:59.643065511 +0000 UTC m=+17.057722495 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:51.744278 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:51.744221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:51.744487 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:51.744288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:51.744487 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.744445 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:51.744626 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.744503 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:56:59.744486143 +0000 UTC m=+17.159143125 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:51.745012 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.744874 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:56:51.745012 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.744904 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:56:51.745012 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.744925 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:51.745012 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:51.744986 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:56:59.744966229 +0000 UTC m=+17.159623303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:52.054051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:52.053056 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:52.054051 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:52.053300 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:52.054051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:52.053704 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:52.054051 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:52.053831 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:52.054051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:52.053884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:52.054051 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:52.053962 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:54.054349 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:54.053562 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:54.054349 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:54.053574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:54.054349 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:54.053958 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:54.054349 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:54.054046 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:54.054349 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:54.053628 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:54.054349 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:54.054155 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:56.053853 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:56.053770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:56.054237 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:56.053770 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:56.054237 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:56.053877 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:56.054237 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:56.053970 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:56.054237 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:56.053791 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:56.054237 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:56.054051 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:58.053440 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:58.053399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:58.053440 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:58.053428 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:58.053975 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:58.053435 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:58.053975 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:58.053520 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:56:58.053975 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:58.053583 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:56:58.053975 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:58.053717 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:56:59.701772 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:59.701719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:56:59.702187 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.701860 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:59.702187 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.701930 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:57:15.701910387 +0000 UTC m=+33.116567380 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:56:59.802598 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:59.802563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:56:59.802598 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:56:59.802604 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:56:59.802809 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.802721 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:59.802809 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.802745 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:56:59.802809 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.802768 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:56:59.802809 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.802774 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:57:15.802761399 +0000 UTC m=+33.217418382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:56:59.802809 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.802779 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:56:59.802977 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:56:59.802821 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:15.802807358 +0000 UTC m=+33.217464339 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:57:00.053598 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:00.053554 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:00.053598 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:00.053553 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:00.053855 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:00.053698 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:00.053855 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:00.053716 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:00.053855 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:00.053809 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:00.054011 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:00.053951 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:02.052873 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:02.052833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:02.053440 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:02.052834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:02.053440 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:02.052976 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:02.053440 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:02.052834 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:02.053440 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:02.053095 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:02.053440 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:02.053180 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:04.053223 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:04.053190 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:04.053223 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:04.053232 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:04.053788 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:04.053303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:04.053788 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:04.053320 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:04.053788 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:04.053419 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:04.053788 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:04.053542 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:05.256892 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.256465 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-mghpz" event={"ID":"a3ba8fda-d419-4c09-880f-8a768f7cd1a0","Type":"ContainerStarted","Data":"313b8d45ea7f1de2b190cc7169500b4d71e1449feff019569acd9ce66c06d57c"} Jul 28 17:57:05.258051 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.258027 2575 generic.go:356] "Generic (PLEG): container finished" podID="5071a057a738bd7dfffd7bae34454265" containerID="1dc0b1aea6234cf19ef9584f7f3412b32e8fd314d73391a0c10230274e5d2174" exitCode=0 Jul 28 17:57:05.258161 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.258114 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" event={"ID":"5071a057a738bd7dfffd7bae34454265","Type":"ContainerDied","Data":"1dc0b1aea6234cf19ef9584f7f3412b32e8fd314d73391a0c10230274e5d2174"} Jul 28 17:57:05.259539 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.259518 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6gtgh" event={"ID":"df10c201-bf85-4f6c-b7b1-e79bc719405c","Type":"ContainerStarted","Data":"3d465f0f8eb40aa251a5d8737da87ecd83fdfbc2877a01231d052870a073b3fd"} Jul 28 17:57:05.262406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.262375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"2f22b623dabf32491a7f64fe4d687f8501fe79c07302dc6b6a5f615a9caadee3"} Jul 28 17:57:05.262547 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.262453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"30a08839f872611ee334d2c4be528e6e55ddaf4e34be5d1f097797f0688bc441"} Jul 28 17:57:05.262547 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.262501 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"d696b3951f7041391962dc0e8f383fc6135a0d3a0157851ecb88f213bb5db3e7"} Jul 28 17:57:05.262547 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.262546 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"a0b0ac64ddbfdb1560aeb0b54c544e6552527072dd7bf461fb81f9e7b4067293"} Jul 28 17:57:05.262733 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.262594 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"741c12c59347c8ffec42f21b0ebe991c2684bd18a7dc4ae6c084e5ae2d887d30"} Jul 28 17:57:05.262733 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.262641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"91f93b8fd234916e71566dfae4b393d03c623a752481dfc5b4d781216b5244cf"} Jul 28 17:57:05.263671 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.263641 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" event={"ID":"0cfc68e8-0838-4804-9704-d95d306e26df","Type":"ContainerStarted","Data":"54739e7601fd2184b47fbcf1f8d179b23237220b38bb1cff0d9dce106a37193b"} Jul 28 17:57:05.265046 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.265023 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" event={"ID":"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9","Type":"ContainerStarted","Data":"d4a83f9324c68950bd2321360ae1176e4c879038cfc34802f2c0d944793f1dce"} Jul 28 17:57:05.266354 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.266333 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae" containerID="9475c8bce4653b8a861c2a1a0e5b96e70231ce0b5ed29df9981ad1ed2eaebeca" exitCode=0 Jul 28 17:57:05.266438 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.266393 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerDied","Data":"9475c8bce4653b8a861c2a1a0e5b96e70231ce0b5ed29df9981ad1ed2eaebeca"} Jul 28 17:57:05.267756 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.267718 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-b44c7" event={"ID":"c88eb666-c963-4a96-a1cf-67097790c4df","Type":"ContainerStarted","Data":"b716debd55aed5f2de6fce29e6601087a2a5121105315c61c95118e81e21b0d2"} Jul 28 17:57:05.269072 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.269048 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" event={"ID":"545c1a00c368df77d9e72312b6e155db","Type":"ContainerStarted","Data":"7c1ed0fe9e508b4719556a4dedde96016d6a681fd1b3ba210d0b26171e46f6b6"} Jul 28 17:57:05.271914 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.271867 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-mghpz" podStartSLOduration=1.514162408 podStartE2EDuration="21.271854658s" totalImagesPullingTime="19.75769225s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.271738623 +0000 UTC m=+22.686395624" watchObservedRunningTime="2026-07-28 17:57:05.271854658 +0000 UTC m=+22.686511660" Jul 28 17:57:05.327485 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.327435 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wcfrz" podStartSLOduration=1.5746274630000001 podStartE2EDuration="21.327418459s" totalImagesPullingTime="19.752790996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.327110412 +0000 UTC m=+22.741767408" watchObservedRunningTime="2026-07-28 17:57:05.327418459 +0000 UTC m=+22.742075460" Jul 28 17:57:05.342551 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.341975 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6gtgh" podStartSLOduration=5.760331657 podStartE2EDuration="21.341951726s" totalImagesPullingTime="15.581620069s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.34193913 +0000 UTC m=+22.756596133" watchObservedRunningTime="2026-07-28 17:57:05.341951726 +0000 UTC m=+22.756608729" Jul 28 17:57:05.368718 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.367710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-147.ec2.internal" podStartSLOduration=21.367657175 podStartE2EDuration="21.367657175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.367305715 +0000 UTC m=+22.781962721" watchObservedRunningTime="2026-07-28 17:57:05.367657175 +0000 UTC m=+22.782314178" Jul 28 17:57:05.389914 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:05.389871 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-b44c7" podStartSLOduration=2.390663618 podStartE2EDuration="22.389853414s" totalImagesPullingTime="19.999189796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:05.389790753 +0000 UTC m=+22.804447755" watchObservedRunningTime="2026-07-28 17:57:05.389853414 +0000 UTC m=+22.804510415" Jul 28 17:57:06.053152 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.053062 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:06.053152 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.053107 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:06.053448 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.053172 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:06.053448 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:06.053183 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:06.053448 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:06.053275 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:06.053448 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:06.053345 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:06.273034 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.272999 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-sg44n" event={"ID":"0e18bfb5-2d47-4cfc-8f64-86b1a23606a1","Type":"ContainerStarted","Data":"df1b2d60d7de631a8bd2de4e4fb08b1ab8fad3c775ee468651d830f1517474c1"} Jul 28 17:57:06.275006 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.274968 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rgqsk" event={"ID":"98740ddb-0e74-4f8a-9e02-615ea70bc398","Type":"ContainerStarted","Data":"6cebe5320666f30330a45631ad73d212eef517c5a729b9364ef8af049fbabe1c"} Jul 28 17:57:06.277142 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.277085 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" event={"ID":"5071a057a738bd7dfffd7bae34454265","Type":"ContainerStarted","Data":"ef12c0047250d43d22e649d9be35f59f26b7995c1f301185dfc480f05523c736"} Jul 28 17:57:06.301033 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.300981 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-sg44n" podStartSLOduration=2.540572341 podStartE2EDuration="22.300961549s" totalImagesPullingTime="19.760389208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:06.287268591 +0000 UTC m=+23.701925595" watchObservedRunningTime="2026-07-28 17:57:06.300961549 +0000 UTC m=+23.715618551" Jul 28 17:57:06.301356 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.301328 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rgqsk" podStartSLOduration=1.132804991 podStartE2EDuration="22.301319247s" totalImagesPullingTime="21.168514256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-28 17:57:06.301106101 +0000 UTC m=+23.715763087" watchObservedRunningTime="2026-07-28 17:57:06.301319247 +0000 UTC m=+23.715976251" Jul 28 17:57:06.316849 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.316790 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-147.ec2.internal" podStartSLOduration=22.316773347 podStartE2EDuration="22.316773347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:06.316287709 +0000 UTC m=+23.730944713" watchObservedRunningTime="2026-07-28 17:57:06.316773347 +0000 UTC m=+23.731430349" Jul 28 17:57:06.445053 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:06.445012 2575 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 17:57:07.033532 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:07.033417 2575 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:57:06.445039758Z","UUID":"9353d391-ea74-499d-8099-2504b680ce30","Handler":null,"Name":"","Endpoint":""} Jul 28 17:57:07.035800 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:07.035493 2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 17:57:07.035800 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:07.035540 2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 17:57:07.281795 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:07.281704 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" event={"ID":"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9","Type":"ContainerStarted","Data":"e2d1343678a3ac304dc0b978c74a43e621150a1c85948d6df19a2a1dfc914edb"} Jul 28 17:57:07.285632 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:07.285524 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"49489b27c4a66138de0688c0db4fead0084ffe9386d7a275810085049a574a35"} Jul 28 17:57:08.053181 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:08.053103 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:08.053372 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:08.053114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:08.053372 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:08.053237 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:08.053372 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:08.053278 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:08.053372 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:08.053114 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:08.053372 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:08.053357 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:08.290172 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:08.290124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" event={"ID":"dc3dfca2-b36c-4613-b67f-1b2aaf7567d9","Type":"ContainerStarted","Data":"f4222c005e83b3b2b411563fb184ac57dc260507ad189b418cd1e6bc69c50b20"} Jul 28 17:57:08.310320 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:08.310216 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7rvpw" podStartSLOduration=3.010162172 podStartE2EDuration="25.310195658s" totalImagesPullingTime="22.300033486s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:08.310085552 +0000 UTC m=+25.724742558" watchObservedRunningTime="2026-07-28 17:57:08.310195658 +0000 UTC m=+25.724852662" Jul 28 17:57:10.046200 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.045936 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:57:10.046891 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.046494 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:57:10.052775 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.052746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:10.052908 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.052746 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:10.052908 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:10.052859 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:10.052991 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.052753 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:10.053023 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:10.052913 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:10.053023 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:10.053009 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:10.298393 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.298287 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" event={"ID":"61ce049e-c323-430a-95c0-fa3618049ab1","Type":"ContainerStarted","Data":"63398c9ffd7de4c5ddd7cd8e79bdde295efd490e1496a2db12c3d09aad76621f"} Jul 28 17:57:10.298561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.298516 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:57:10.300195 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.300170 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae" containerID="ed4b028d3094ab296988dab5e30517d9c655f7612896f0e02bea6efacb4f79fd" exitCode=0 Jul 28 17:57:10.300337 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.300255 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerDied","Data":"ed4b028d3094ab296988dab5e30517d9c655f7612896f0e02bea6efacb4f79fd"} Jul 28 17:57:10.300446 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.300428 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:57:10.301234 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.300883 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6gtgh" Jul 28 17:57:10.314268 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.314234 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:57:10.327441 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:10.327391 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" podStartSLOduration=6.577578004 podStartE2EDuration="26.32737227s" totalImagesPullingTime="19.749794266s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:10.327262633 +0000 UTC m=+27.741919635" watchObservedRunningTime="2026-07-28 17:57:10.32737227 +0000 UTC m=+27.742029271" Jul 28 17:57:11.303074 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.302892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:57:11.303486 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.303136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:57:11.318370 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.318343 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:57:11.521956 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.521838 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ztp68"] Jul 28 17:57:11.522103 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.521960 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:11.522103 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:11.522050 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:11.524995 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.524971 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fjvtt"] Jul 28 17:57:11.525133 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.525078 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:11.525193 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:11.525165 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:11.525474 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.525457 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qsvk"] Jul 28 17:57:11.525553 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:11.525541 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:11.525625 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:11.525608 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:12.309500 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:12.309461 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae" containerID="f80580a97bb1e60a52705594e109669c8e42f82f633ee483404115a886cd8834" exitCode=0 Jul 28 17:57:12.309932 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:12.309547 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerDied","Data":"f80580a97bb1e60a52705594e109669c8e42f82f633ee483404115a886cd8834"} Jul 28 17:57:13.055843 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:13.055815 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:13.055843 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:13.055849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:13.056060 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:13.055898 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:13.056060 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:13.055919 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:13.056060 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:13.055986 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:13.056060 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:13.056040 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:14.315617 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:14.315576 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae" containerID="17239379e6ead3d51b33ae06011ea19b9e7d4e4e78bca49ebdc95729713a5806" exitCode=0 Jul 28 17:57:14.316049 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:14.315638 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerDied","Data":"17239379e6ead3d51b33ae06011ea19b9e7d4e4e78bca49ebdc95729713a5806"} Jul 28 17:57:15.053375 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:15.053336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:15.053561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:15.053336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:15.053561 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.053459 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-2qsvk" podUID="ff963ef2-56e2-43f2-8977-7854eb82f754" Jul 28 17:57:15.053650 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.053554 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fjvtt" podUID="144d01ab-98a5-4f4c-bee9-a4030a422bec" Jul 28 17:57:15.053650 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:15.053336 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:15.053777 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.053648 2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-ztp68" podUID="63e15748-3635-4ac4-be53-09c308ba90bb" Jul 28 17:57:15.726156 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:15.726112 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:15.726585 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.726261 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:57:15.726585 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.726336 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:57:47.726315893 +0000 UTC m=+65.140972895 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 17:57:15.826820 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:15.826775 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:15.826995 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:15.826832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:15.826995 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.826911 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 17:57:15.826995 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.826932 2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 17:57:15.826995 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.826945 2575 projected.go:196] Error preparing data for projected volume kube-api-access-2v6sb for pod openshift-network-diagnostics/network-check-target-2qsvk: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:57:15.827214 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.826994 2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 17:57:15.827214 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.827004 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb podName:ff963ef2-56e2-43f2-8977-7854eb82f754 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:47.826988672 +0000 UTC m=+65.241645656 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2v6sb" (UniqueName: "kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb") pod "network-check-target-2qsvk" (UID: "ff963ef2-56e2-43f2-8977-7854eb82f754") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 17:57:15.827214 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:15.827067 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret podName:63e15748-3635-4ac4-be53-09c308ba90bb nodeName:}" failed. No retries permitted until 2026-07-28 17:57:47.827048003 +0000 UTC m=+65.241704990 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret") pod "global-pull-secret-syncer-ztp68" (UID: "63e15748-3635-4ac4-be53-09c308ba90bb") : object "kube-system"/"original-pull-secret" not registered Jul 28 17:57:16.863019 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.862983 2575 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-147.ec2.internal" event="NodeReady" Jul 28 17:57:16.863455 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.863156 2575 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 17:57:16.899625 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.899592 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-57cf8df6dc-bm9zl"] Jul 28 17:57:16.902052 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.902025 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:16.905238 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.905212 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vrsc6\"" Jul 28 17:57:16.905379 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.905361 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 28 17:57:16.905492 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.905471 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 28 17:57:16.905578 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.905496 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 28 17:57:16.912154 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.912128 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6ddcz"] Jul 28 17:57:16.912273 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.912190 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 28 17:57:16.914104 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.914083 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:16.916221 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.916196 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kwltp"] Jul 28 17:57:16.916641 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.916361 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 17:57:16.916641 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.916446 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mqwj\"" Jul 28 17:57:16.916641 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.916455 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 17:57:16.918057 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.918038 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57cf8df6dc-bm9zl"] Jul 28 17:57:16.918174 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.918140 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:16.921392 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.920995 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 17:57:16.921392 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.921026 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 17:57:16.921392 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.921209 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vjgn2\"" Jul 28 17:57:16.921392 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.921214 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 17:57:16.928790 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.928768 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kwltp"] Jul 28 17:57:16.932132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:16.932090 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6ddcz"] Jul 28 17:57:17.039177 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039177 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-installation-pull-secrets\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039177 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039143 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rggm\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-kube-api-access-2rggm\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039191 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-bound-sa-token\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039258 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5cfce10-c28f-463f-ab5e-0626d77afe0d-cert\") pod \"ingress-canary-kwltp\" (UID: \"b5cfce10-c28f-463f-ab5e-0626d77afe0d\") " pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039290 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-registry-certificates\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm4rw\" (UniqueName: \"kubernetes.io/projected/b5cfce10-c28f-463f-ab5e-0626d77afe0d-kube-api-access-hm4rw\") pod \"ingress-canary-kwltp\" (UID: \"b5cfce10-c28f-463f-ab5e-0626d77afe0d\") " pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039370 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-tmp-dir\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.039458 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039433 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-config-volume\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.039782 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039480 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-image-registry-private-configuration\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039782 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039509 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-trusted-ca\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039782 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039596 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29410269-90df-41a5-adcf-7d5c3136a321-ca-trust-extracted\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.039782 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.039672 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd82f\" (UniqueName: \"kubernetes.io/projected/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-kube-api-access-sd82f\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.053185 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.053145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:17.053363 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.053145 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:17.053363 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.053276 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:17.056073 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.056049 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:57:17.056459 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.056294 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:57:17.056459 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.056322 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pbl4v\"" Jul 28 17:57:17.056459 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.056378 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:57:17.056707 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.056378 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sgwbs\"" Jul 28 17:57:17.056707 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.056551 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:57:17.084148 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.084114 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:17.141065 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141027 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-tmp-dir\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.141258 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141079 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-config-volume\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.141258 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141221 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-image-registry-private-configuration\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141369 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141266 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-trusted-ca\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141369 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141314 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29410269-90df-41a5-adcf-7d5c3136a321-ca-trust-extracted\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141463 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd82f\" (UniqueName: \"kubernetes.io/projected/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-kube-api-access-sd82f\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.141463 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141411 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141463 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141437 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-installation-pull-secrets\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141609 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141461 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rggm\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-kube-api-access-2rggm\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141609 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-bound-sa-token\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141609 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141536 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5cfce10-c28f-463f-ab5e-0626d77afe0d-cert\") pod \"ingress-canary-kwltp\" (UID: \"b5cfce10-c28f-463f-ab5e-0626d77afe0d\") " pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.141609 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141566 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-registry-certificates\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.141609 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141589 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.141862 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141618 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm4rw\" (UniqueName: \"kubernetes.io/projected/b5cfce10-c28f-463f-ab5e-0626d77afe0d-kube-api-access-hm4rw\") pod \"ingress-canary-kwltp\" (UID: \"b5cfce10-c28f-463f-ab5e-0626d77afe0d\") " pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.141862 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.141749 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-tmp-dir\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.142268 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.142045 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-config-volume\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.142386 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.142065 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:17.142386 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.142300 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:17.142386 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.142358 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:17.642339083 +0000 UTC m=+35.056996076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:17.142549 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.142454 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-trusted-ca\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.142549 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.142130 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29410269-90df-41a5-adcf-7d5c3136a321-ca-trust-extracted\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.142634 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.142556 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:17.142634 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.142611 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:17.642587048 +0000 UTC m=+35.057244040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:17.142802 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.142776 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-registry-certificates\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.146178 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.146081 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-installation-pull-secrets\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.146178 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.146090 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-image-registry-private-configuration\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.146178 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.146137 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5cfce10-c28f-463f-ab5e-0626d77afe0d-cert\") pod \"ingress-canary-kwltp\" (UID: \"b5cfce10-c28f-463f-ab5e-0626d77afe0d\") " pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.151297 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.151254 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-bound-sa-token\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.151875 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.151850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm4rw\" (UniqueName: \"kubernetes.io/projected/b5cfce10-c28f-463f-ab5e-0626d77afe0d-kube-api-access-hm4rw\") pod \"ingress-canary-kwltp\" (UID: \"b5cfce10-c28f-463f-ab5e-0626d77afe0d\") " pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.155191 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.155157 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd82f\" (UniqueName: \"kubernetes.io/projected/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-kube-api-access-sd82f\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.155617 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.155595 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rggm\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-kube-api-access-2rggm\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.185410 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.185357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:17.236335 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.236291 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kwltp" Jul 28 17:57:17.647196 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.647146 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:17.647381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:17.647254 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:17.647381 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.647329 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:17.647381 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.647358 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:17.647381 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.647369 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:17.647532 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.647395 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:18.647378915 +0000 UTC m=+36.062035895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:17.647532 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:17.647416 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:18.64740453 +0000 UTC m=+36.062061513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:18.344109 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:18.344081 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kwltp"] Jul 28 17:57:18.349157 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:57:18.349131 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cfce10_c28f_463f_ab5e_0626d77afe0d.slice/crio-ed7665e6d6c9b5fb8cd84d73b134390cecf5d932faedaa8d30e09cd0b3f7bd2d WatchSource:0}: Error finding container ed7665e6d6c9b5fb8cd84d73b134390cecf5d932faedaa8d30e09cd0b3f7bd2d: Status 404 returned error can't find the container with id ed7665e6d6c9b5fb8cd84d73b134390cecf5d932faedaa8d30e09cd0b3f7bd2d Jul 28 17:57:18.662986 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:18.662788 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:18.663165 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:18.663013 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:18.663165 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:18.662934 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:18.663165 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:18.663063 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:18.663165 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:18.663104 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:18.663165 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:18.663117 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:20.663102861 +0000 UTC m=+38.077759841 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:18.663165 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:18.663142 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:20.663131254 +0000 UTC m=+38.077788238 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:19.329316 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:19.329272 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kwltp" event={"ID":"b5cfce10-c28f-463f-ab5e-0626d77afe0d","Type":"ContainerStarted","Data":"ed7665e6d6c9b5fb8cd84d73b134390cecf5d932faedaa8d30e09cd0b3f7bd2d"} Jul 28 17:57:19.332586 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:19.332554 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae" containerID="45f28ee7f8fcf4309b7e64704ce73272c6c261be246115f574206d32d82e264d" exitCode=0 Jul 28 17:57:19.332782 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:19.332597 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerDied","Data":"45f28ee7f8fcf4309b7e64704ce73272c6c261be246115f574206d32d82e264d"} Jul 28 17:57:20.339487 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:20.339448 2575 generic.go:356] "Generic (PLEG): container finished" podID="9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae" containerID="e28d6b6ab2c31d65b2cf1405d283ddd812f8152a0f396e45ad9350fe147137e5" exitCode=0 Jul 28 17:57:20.339980 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:20.339509 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerDied","Data":"e28d6b6ab2c31d65b2cf1405d283ddd812f8152a0f396e45ad9350fe147137e5"} Jul 28 17:57:20.679143 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:20.679106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:20.679304 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:20.679208 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:20.679304 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:20.679265 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:20.679423 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:20.679335 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:24.679314579 +0000 UTC m=+42.093971574 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:20.679423 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:20.679373 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:20.679423 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:20.679399 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:20.679538 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:20.679448 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:24.679433439 +0000 UTC m=+42.094090436 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:21.345963 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:21.345922 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" event={"ID":"9c0b6a3f-dcd9-48bf-8cb0-9c42e2de7dae","Type":"ContainerStarted","Data":"23ddd84a5a94c99f6280500c8f4e149badf9151516c81ebde3a885ffb68acb54"} Jul 28 17:57:21.370917 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:21.370858 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qrp7l" podStartSLOduration=8.194054988 podStartE2EDuration="38.370837198s" totalImagesPullingTime="30.17678221s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:21.36964562 +0000 UTC m=+38.784302622" watchObservedRunningTime="2026-07-28 17:57:21.370837198 +0000 UTC m=+38.785494200" Jul 28 17:57:21.382541 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:21.382510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:21.587975 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:21.587877 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:21.672406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:21.672368 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:22.349479 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:22.349442 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kwltp" event={"ID":"b5cfce10-c28f-463f-ab5e-0626d77afe0d","Type":"ContainerStarted","Data":"8547c7a29e8670bd0f4997fa15c88ba24fa67b52b7f5697e95d076ed195305ec"} Jul 28 17:57:22.364795 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:22.364741 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kwltp" podStartSLOduration=3.45609667 podStartE2EDuration="6.364722955s" totalImagesPullingTime="2.908626285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:22.36436984 +0000 UTC m=+39.779026841" watchObservedRunningTime="2026-07-28 17:57:22.364722955 +0000 UTC m=+39.779379956" Jul 28 17:57:23.301352 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:23.301312 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:23.372452 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:23.372415 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:24.224027 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:24.223996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:24.299701 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:24.299649 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:24.709519 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:24.709478 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:24.709993 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:24.709534 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:24.709993 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:24.709641 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:24.709993 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:24.709664 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:24.709993 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:24.709708 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:24.709993 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:24.709717 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:32.709698875 +0000 UTC m=+50.124355868 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:24.709993 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:24.709768 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:32.709749829 +0000 UTC m=+50.124406830 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:27.689931 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:27.689897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:29.836058 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:29.835959 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:29.905605 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:29.905566 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:30.792046 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:30.792012 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:30.860034 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:30.860002 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:32.771293 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:32.771250 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:32.771772 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:32.771306 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:32.771772 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:32.771394 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:32.771772 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:32.771419 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:32.771772 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:32.771403 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:32.771772 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:32.771475 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:48.771459622 +0000 UTC m=+66.186116603 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:32.771772 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:32.771489 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:48.771482269 +0000 UTC m=+66.186139249 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:33.107412 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:33.107326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:33.208833 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:33.208802 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:35.459358 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:35.459317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:35.516853 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:35.516819 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:39.305235 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:39.305201 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:39.403179 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:39.403138 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:42.378988 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:42.376764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:42.481571 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:42.481539 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:43.325927 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:43.325898 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2ck7j" Jul 28 17:57:43.484474 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:43.484439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:43.582122 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:43.582035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:45.886744 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:45.886710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:45.987178 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:45.987150 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:46.590817 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:46.590784 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:46.673987 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:46.673945 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:47.778540 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.778504 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:57:47.781223 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.781202 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 17:57:47.789340 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:47.789315 2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 17:57:47.789404 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:47.789393 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs podName:144d01ab-98a5-4f4c-bee9-a4030a422bec nodeName:}" failed. No retries permitted until 2026-07-28 17:58:51.789376814 +0000 UTC m=+129.204033794 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs") pod "network-metrics-daemon-fjvtt" (UID: "144d01ab-98a5-4f4c-bee9-a4030a422bec") : secret "metrics-daemon-secret" not found Jul 28 17:57:47.879512 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.879472 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:47.879601 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.879520 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:47.882142 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.882122 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 17:57:47.882180 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.882124 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 17:57:47.892769 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.892748 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 17:57:47.893632 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.893604 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/63e15748-3635-4ac4-be53-09c308ba90bb-original-pull-secret\") pod \"global-pull-secret-syncer-ztp68\" (UID: \"63e15748-3635-4ac4-be53-09c308ba90bb\") " pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:47.903339 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.903312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2v6sb\" (UniqueName: \"kubernetes.io/projected/ff963ef2-56e2-43f2-8977-7854eb82f754-kube-api-access-2v6sb\") pod \"network-check-target-2qsvk\" (UID: \"ff963ef2-56e2-43f2-8977-7854eb82f754\") " pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:47.966822 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.966789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sgwbs\"" Jul 28 17:57:47.971765 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.971742 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-ztp68" Jul 28 17:57:47.974485 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:47.974467 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:48.104603 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.104576 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-2qsvk"] Jul 28 17:57:48.108433 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:57:48.108408 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff963ef2_56e2_43f2_8977_7854eb82f754.slice/crio-fab896d5d017cad102aa90d2535ab08881b2c7609bc6ca03465e4981a52e1a68 WatchSource:0}: Error finding container fab896d5d017cad102aa90d2535ab08881b2c7609bc6ca03465e4981a52e1a68: Status 404 returned error can't find the container with id fab896d5d017cad102aa90d2535ab08881b2c7609bc6ca03465e4981a52e1a68 Jul 28 17:57:48.118097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.118070 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-ztp68"] Jul 28 17:57:48.120991 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:57:48.120965 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e15748_3635_4ac4_be53_09c308ba90bb.slice/crio-956e00e083138f697202d793a2b8d854720d86bee479e7bc683b460f6690e6d8 WatchSource:0}: Error finding container 956e00e083138f697202d793a2b8d854720d86bee479e7bc683b460f6690e6d8: Status 404 returned error can't find the container with id 956e00e083138f697202d793a2b8d854720d86bee479e7bc683b460f6690e6d8 Jul 28 17:57:48.410443 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.410406 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qsvk" event={"ID":"ff963ef2-56e2-43f2-8977-7854eb82f754","Type":"ContainerStarted","Data":"fab896d5d017cad102aa90d2535ab08881b2c7609bc6ca03465e4981a52e1a68"} Jul 28 17:57:48.411352 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.411330 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ztp68" event={"ID":"63e15748-3635-4ac4-be53-09c308ba90bb","Type":"ContainerStarted","Data":"956e00e083138f697202d793a2b8d854720d86bee479e7bc683b460f6690e6d8"} Jul 28 17:57:48.575678 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.575599 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv"] Jul 28 17:57:48.578154 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.578122 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" Jul 28 17:57:48.582015 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.581995 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 28 17:57:48.582175 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.582152 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 28 17:57:48.582275 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.582201 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-qc6jg\"" Jul 28 17:57:48.590527 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.590503 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv"] Jul 28 17:57:48.686774 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.686667 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bb2\" (UniqueName: \"kubernetes.io/projected/96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6-kube-api-access-78bb2\") pod \"volume-data-source-validator-549c6d54cf-2mpqv\" (UID: \"96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" Jul 28 17:57:48.759202 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.759168 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:48.787334 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.787295 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.787346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bb2\" (UniqueName: \"kubernetes.io/projected/96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6-kube-api-access-78bb2\") pod \"volume-data-source-validator-549c6d54cf-2mpqv\" (UID: \"96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.787400 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:48.787467 2575 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:48.787486 2575 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-57cf8df6dc-bm9zl: secret "image-registry-tls" not found Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:48.787506 2575 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:48.787545 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls podName:29410269-90df-41a5-adcf-7d5c3136a321 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:20.787522706 +0000 UTC m=+98.202179705 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls") pod "image-registry-57cf8df6dc-bm9zl" (UID: "29410269-90df-41a5-adcf-7d5c3136a321") : secret "image-registry-tls" not found Jul 28 17:57:48.787822 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:48.787571 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls podName:8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:20.78755848 +0000 UTC m=+98.202215464 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls") pod "dns-default-6ddcz" (UID: "8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0") : secret "dns-default-metrics-tls" not found Jul 28 17:57:48.804772 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.804738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bb2\" (UniqueName: \"kubernetes.io/projected/96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6-kube-api-access-78bb2\") pod \"volume-data-source-validator-549c6d54cf-2mpqv\" (UID: \"96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" Jul 28 17:57:48.860379 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.860327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:48.887325 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:48.887283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" Jul 28 17:57:49.045393 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:49.045359 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv"] Jul 28 17:57:49.049392 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:57:49.049362 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e9ed9f_ec57_4493_9bdf_5f86ee6d54b6.slice/crio-e9cb1e349f0aa636dd24348d64f33b0cb4e4e5cad09b138c214e395430dadf88 WatchSource:0}: Error finding container e9cb1e349f0aa636dd24348d64f33b0cb4e4e5cad09b138c214e395430dadf88: Status 404 returned error can't find the container with id e9cb1e349f0aa636dd24348d64f33b0cb4e4e5cad09b138c214e395430dadf88 Jul 28 17:57:49.415919 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:49.415880 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" event={"ID":"96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6","Type":"ContainerStarted","Data":"e9cb1e349f0aa636dd24348d64f33b0cb4e4e5cad09b138c214e395430dadf88"} Jul 28 17:57:53.428126 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.428092 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-2qsvk" event={"ID":"ff963ef2-56e2-43f2-8977-7854eb82f754","Type":"ContainerStarted","Data":"5106f0a2ab0eaeebda331e7d54f1a11e46896d88778df0d9f63bfb2b19f8c3c8"} Jul 28 17:57:53.428590 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.428173 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:57:53.429614 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.429593 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-ztp68" event={"ID":"63e15748-3635-4ac4-be53-09c308ba90bb","Type":"ContainerStarted","Data":"7328fecc3055867170b7f343892a52f74b9ae9408652007ebc8aa41f51967883"} Jul 28 17:57:53.430922 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.430897 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" event={"ID":"96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6","Type":"ContainerStarted","Data":"dbf99a272a1f5f7d6d874387e283d9d1b6c92ab0f990fa36768ab68af3e5a0c0"} Jul 28 17:57:53.446634 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.444586 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-2qsvk" podStartSLOduration=65.417774417 podStartE2EDuration="1m10.444564578s" totalImagesPullingTime="5.026790161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:53.443856201 +0000 UTC m=+70.858513207" watchObservedRunningTime="2026-07-28 17:57:53.444564578 +0000 UTC m=+70.859221587" Jul 28 17:57:53.459547 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.459479 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-2mpqv" podStartSLOduration=1.385714345 podStartE2EDuration="5.459461043s" totalImagesPullingTime="4.073746698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:53.459008335 +0000 UTC m=+70.873665338" watchObservedRunningTime="2026-07-28 17:57:53.459461043 +0000 UTC m=+70.874118078" Jul 28 17:57:53.473182 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.473097 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-ztp68" podStartSLOduration=64.458861665 podStartE2EDuration="1m9.473080668s" totalImagesPullingTime="5.014219003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:57:53.472588422 +0000 UTC m=+70.887245436" watchObservedRunningTime="2026-07-28 17:57:53.473080668 +0000 UTC m=+70.887737669" Jul 28 17:57:53.562900 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.562864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:53.659723 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:53.659666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:54.078818 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:54.078783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:54.163360 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:54.163325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:54.238111 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:54.238073 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:54.778909 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:54.778872 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:54.841477 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:54.841437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:55.163852 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.163822 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l9pkg"] Jul 28 17:57:55.165835 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.165820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.172362 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.172335 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 17:57:55.172607 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.172590 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 17:57:55.172722 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.172631 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 17:57:55.173475 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.173455 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q7kkv\"" Jul 28 17:57:55.173475 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.173452 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 17:57:55.179838 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.179806 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9pkg"] Jul 28 17:57:55.215302 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.215269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:55.242823 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.242786 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.243003 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.242850 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvwjb\" (UniqueName: \"kubernetes.io/projected/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-api-access-qvwjb\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.243003 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.242952 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.243113 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.243009 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c0509b-69ea-4443-b397-6e7fb7313c19-crio-socket\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.243113 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.243058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c0509b-69ea-4443-b397-6e7fb7313c19-data-volume\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.343978 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.343940 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c0509b-69ea-4443-b397-6e7fb7313c19-crio-socket\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.344106 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.343992 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c0509b-69ea-4443-b397-6e7fb7313c19-data-volume\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.344106 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.344022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.344106 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.344056 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvwjb\" (UniqueName: \"kubernetes.io/projected/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-api-access-qvwjb\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.344106 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.344082 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.344262 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.344133 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:57:55.344262 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.344151 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:57:55.344262 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.344164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/51c0509b-69ea-4443-b397-6e7fb7313c19-crio-socket\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.344262 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.344208 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:55.844193598 +0000 UTC m=+73.258850579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : configmap "kube-rbac-proxy" not found Jul 28 17:57:55.344262 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.344224 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:55.844216833 +0000 UTC m=+73.258873813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : secret "insights-runtime-extractor-tls" not found Jul 28 17:57:55.344524 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.344373 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/51c0509b-69ea-4443-b397-6e7fb7313c19-data-volume\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.352204 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.352180 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvwjb\" (UniqueName: \"kubernetes.io/projected/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-api-access-qvwjb\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.552483 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.552393 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:55.651898 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.651866 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:55.848126 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.848010 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.848126 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:55.848119 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:55.848579 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.848140 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:57:55.848579 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.848209 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:56.848192614 +0000 UTC m=+74.262849597 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : configmap "kube-rbac-proxy" not found Jul 28 17:57:55.848579 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.848233 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:57:55.848579 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:55.848310 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:56.848292453 +0000 UTC m=+74.262949434 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : secret "insights-runtime-extractor-tls" not found Jul 28 17:57:56.350703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:56.350658 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:56.422049 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:56.422022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:56.856299 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:56.856258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:56.856678 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:56.856321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:56.856678 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:56.856408 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:57:56.856678 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:56.856426 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:57:56.856678 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:56.856481 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:58.856464447 +0000 UTC m=+76.271121426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : secret "insights-runtime-extractor-tls" not found Jul 28 17:57:56.856678 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:56.856500 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:57:58.856493429 +0000 UTC m=+76.271150409 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : configmap "kube-rbac-proxy" not found Jul 28 17:57:57.073813 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.073781 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mfjm6"] Jul 28 17:57:57.075930 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.075912 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.078312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.078281 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 28 17:57:57.079221 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.079196 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 28 17:57:57.079342 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.079278 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 28 17:57:57.079342 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.079284 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-98245\"" Jul 28 17:57:57.079342 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.079307 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 28 17:57:57.087542 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.087520 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mfjm6"] Jul 28 17:57:57.117161 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.117070 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:57.158573 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.158528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0bf6a58-4794-49c2-8f34-184ae5681dfb-signing-cabundle\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.158573 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.158574 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0bf6a58-4794-49c2-8f34-184ae5681dfb-signing-key\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.158791 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.158664 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd8dp\" (UniqueName: \"kubernetes.io/projected/e0bf6a58-4794-49c2-8f34-184ae5681dfb-kube-api-access-xd8dp\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.179745 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.179709 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:57.259777 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.259737 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0bf6a58-4794-49c2-8f34-184ae5681dfb-signing-cabundle\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.259969 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.259791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0bf6a58-4794-49c2-8f34-184ae5681dfb-signing-key\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.259969 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.259832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd8dp\" (UniqueName: \"kubernetes.io/projected/e0bf6a58-4794-49c2-8f34-184ae5681dfb-kube-api-access-xd8dp\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.260390 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.260368 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/e0bf6a58-4794-49c2-8f34-184ae5681dfb-signing-cabundle\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.262227 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.262206 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/e0bf6a58-4794-49c2-8f34-184ae5681dfb-signing-key\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.268599 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.268572 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd8dp\" (UniqueName: \"kubernetes.io/projected/e0bf6a58-4794-49c2-8f34-184ae5681dfb-kube-api-access-xd8dp\") pod \"service-ca-658d7b7c78-mfjm6\" (UID: \"e0bf6a58-4794-49c2-8f34-184ae5681dfb\") " pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.384644 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.384615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" Jul 28 17:57:57.507667 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:57.507626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-mfjm6"] Jul 28 17:57:57.512131 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:57:57.512101 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bf6a58_4794_49c2_8f34_184ae5681dfb.slice/crio-11ccbcd8fbe2b16826daa6cafe498685ee967188067bc0a69dffb339e9de78f5 WatchSource:0}: Error finding container 11ccbcd8fbe2b16826daa6cafe498685ee967188067bc0a69dffb339e9de78f5: Status 404 returned error can't find the container with id 11ccbcd8fbe2b16826daa6cafe498685ee967188067bc0a69dffb339e9de78f5 Jul 28 17:57:58.049062 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:58.049039 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mghpz_a3ba8fda-d419-4c09-880f-8a768f7cd1a0/dns-node-resolver/0.log" Jul 28 17:57:58.378936 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:58.378905 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:57:58.446377 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:58.446320 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" event={"ID":"e0bf6a58-4794-49c2-8f34-184ae5681dfb","Type":"ContainerStarted","Data":"11ccbcd8fbe2b16826daa6cafe498685ee967188067bc0a69dffb339e9de78f5"} Jul 28 17:57:58.871547 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:58.871464 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:58.871547 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:58.871538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:57:58.871867 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:58.871654 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:57:58.871867 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:58.871708 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:57:58.871867 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:58.871739 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:02.871720734 +0000 UTC m=+80.286377722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : secret "insights-runtime-extractor-tls" not found Jul 28 17:57:58.871867 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:57:58.871799 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:02.871768628 +0000 UTC m=+80.286425619 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : configmap "kube-rbac-proxy" not found Jul 28 17:57:59.058196 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:59.058172 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rgqsk_98740ddb-0e74-4f8a-9e02-615ea70bc398/node-ca/0.log" Jul 28 17:57:59.847256 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:57:59.847176 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kwltp_b5cfce10-c28f-463f-ab5e-0626d77afe0d/serve-healthcheck-canary/0.log" Jul 28 17:58:00.415089 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.415050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:00.454729 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.454358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" event={"ID":"e0bf6a58-4794-49c2-8f34-184ae5681dfb","Type":"ContainerStarted","Data":"21ec73f53eeecf22ee0a235851220878ef6e244c71fcb76f0dc9cfcc713fa67a"} Jul 28 17:58:00.473533 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.473463 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-mfjm6" podStartSLOduration=1.4484988890000001 podStartE2EDuration="3.473442773s" totalImagesPullingTime="2.024943884s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:00.472034691 +0000 UTC m=+77.886691685" watchObservedRunningTime="2026-07-28 17:58:00.473442773 +0000 UTC m=+77.888099771" Jul 28 17:58:00.569077 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.569036 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:00.670493 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.670404 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:00.852449 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.852416 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:00.955760 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:00.955660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:01.117858 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:01.117812 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:01.952089 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:01.952052 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:02.031533 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:02.031498 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:02.308403 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:02.308319 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:02.717447 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:02.717406 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:02.907576 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:02.907521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:02.907821 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:02.907659 2575 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 17:58:02.907821 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:02.907720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:02.907821 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:02.907758 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:10.907738649 +0000 UTC m=+88.322395649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : configmap "kube-rbac-proxy" not found Jul 28 17:58:02.907821 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:02.907819 2575 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 17:58:02.908042 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:02.907864 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls podName:51c0509b-69ea-4443-b397-6e7fb7313c19 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:10.90785204 +0000 UTC m=+88.322509026 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l9pkg" (UID: "51c0509b-69ea-4443-b397-6e7fb7313c19") : secret "insights-runtime-extractor-tls" not found Jul 28 17:58:03.124553 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:03.124455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:03.316307 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:03.316250 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:03.708834 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:03.708798 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:04.113791 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:04.113706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:04.226264 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:04.226224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:04.326596 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:04.326544 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:04.717253 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:04.717212 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:04.920183 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:04.920059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:04.998281 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:04.998204 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:05.514746 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:05.514706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:06.111861 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:06.111823 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:06.225434 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:06.225402 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:06.326134 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:06.326095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:06.515615 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:06.515585 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:06.913134 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:06.913102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:07.228597 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:07.228495 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:07.514625 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:07.514532 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:07.708866 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:07.708830 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:08.157033 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:08.156999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:08.711540 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:08.711508 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:09.312736 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:09.312702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:09.511552 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:09.511510 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:09.639292 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:09.639261 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:09.740448 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:09.740414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:10.116408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.116324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:10.525099 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.525060 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:10.957589 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.957560 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:10.978607 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.978572 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:10.978770 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.978664 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:10.979141 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.979119 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/51c0509b-69ea-4443-b397-6e7fb7313c19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:10.980892 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:10.980871 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/51c0509b-69ea-4443-b397-6e7fb7313c19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l9pkg\" (UID: \"51c0509b-69ea-4443-b397-6e7fb7313c19\") " pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:11.074356 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:11.074320 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l9pkg" Jul 28 17:58:11.119896 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:11.119853 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:11.198744 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:11.198667 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l9pkg"] Jul 28 17:58:11.201815 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:11.201785 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51c0509b_69ea_4443_b397_6e7fb7313c19.slice/crio-19ee7faf9133769362018edc12bb18c9234ea21c72bd7c6fb5d79333bb520c9b WatchSource:0}: Error finding container 19ee7faf9133769362018edc12bb18c9234ea21c72bd7c6fb5d79333bb520c9b: Status 404 returned error can't find the container with id 19ee7faf9133769362018edc12bb18c9234ea21c72bd7c6fb5d79333bb520c9b Jul 28 17:58:11.484329 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:11.484299 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9pkg" event={"ID":"51c0509b-69ea-4443-b397-6e7fb7313c19","Type":"ContainerStarted","Data":"19ee7faf9133769362018edc12bb18c9234ea21c72bd7c6fb5d79333bb520c9b"} Jul 28 17:58:12.156919 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.156822 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:12.257101 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.257040 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:12.357843 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.357804 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:12.459201 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.459105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:12.657970 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.657922 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:12.824549 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.824458 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:12.902968 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:12.902914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:13.666593 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:13.666559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:13.767449 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:13.767413 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:14.038565 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:14.038474 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:14.250193 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:14.250118 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:14.381755 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:14.381637 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:15.671001 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:15.670957 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:15.739189 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:15.739149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:16.501908 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:16.501859 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9pkg" event={"ID":"51c0509b-69ea-4443-b397-6e7fb7313c19","Type":"ContainerStarted","Data":"87029bc6e462bae70007351657ae01eb40dfb7a824efc33ba08b9096c3f9316c"} Jul 28 17:58:18.110768 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.110733 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:18.210481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.210445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:18.506417 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.506379 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj"] Jul 28 17:58:18.522431 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.522391 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.525890 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.525865 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 28 17:58:18.529390 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.529350 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 28 17:58:18.530545 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.530349 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 28 17:58:18.534177 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.534158 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 28 17:58:18.543608 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.543576 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj"] Jul 28 17:58:18.639666 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.639626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1aa4c24-1149-41da-8bf8-e972ca19a17f-tmp\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.639666 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.639670 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1aa4c24-1149-41da-8bf8-e972ca19a17f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.639924 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.639789 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gdz\" (UniqueName: \"kubernetes.io/projected/b1aa4c24-1149-41da-8bf8-e972ca19a17f-kube-api-access-q5gdz\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.669447 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.669411 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-chwbs"] Jul 28 17:58:18.674995 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.674965 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:18.679336 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.679311 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 17:58:18.679495 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.679452 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-j7fcw\"" Jul 28 17:58:18.679557 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.679527 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 17:58:18.696488 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.696452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-chwbs"] Jul 28 17:58:18.740996 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.740960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1aa4c24-1149-41da-8bf8-e972ca19a17f-tmp\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.740996 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.740996 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1aa4c24-1149-41da-8bf8-e972ca19a17f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.741286 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.741028 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gdz\" (UniqueName: \"kubernetes.io/projected/b1aa4c24-1149-41da-8bf8-e972ca19a17f-kube-api-access-q5gdz\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.741359 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.741338 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b1aa4c24-1149-41da-8bf8-e972ca19a17f-tmp\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.743502 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.743476 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/b1aa4c24-1149-41da-8bf8-e972ca19a17f-klusterlet-config\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.752739 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.752706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:18.772717 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.772614 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gdz\" (UniqueName: \"kubernetes.io/projected/b1aa4c24-1149-41da-8bf8-e972ca19a17f-kube-api-access-q5gdz\") pod \"klusterlet-addon-workmgr-7947cb57df-6bglj\" (UID: \"b1aa4c24-1149-41da-8bf8-e972ca19a17f\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.832935 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.832895 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:18.842118 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.842078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8qj7\" (UniqueName: \"kubernetes.io/projected/8495b74a-e4fa-4aee-a599-a46e993f7455-kube-api-access-x8qj7\") pod \"downloads-7c97dc4555-chwbs\" (UID: \"8495b74a-e4fa-4aee-a599-a46e993f7455\") " pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:18.849448 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.849414 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:18.943597 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.943558 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8qj7\" (UniqueName: \"kubernetes.io/projected/8495b74a-e4fa-4aee-a599-a46e993f7455-kube-api-access-x8qj7\") pod \"downloads-7c97dc4555-chwbs\" (UID: \"8495b74a-e4fa-4aee-a599-a46e993f7455\") " pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:18.959661 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.959630 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8qj7\" (UniqueName: \"kubernetes.io/projected/8495b74a-e4fa-4aee-a599-a46e993f7455-kube-api-access-x8qj7\") pod \"downloads-7c97dc4555-chwbs\" (UID: \"8495b74a-e4fa-4aee-a599-a46e993f7455\") " pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:18.980241 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.980206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:18.984149 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:18.984120 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:19.067950 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:19.067862 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:19.517662 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:19.517621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9pkg" event={"ID":"51c0509b-69ea-4443-b397-6e7fb7313c19","Type":"ContainerStarted","Data":"3aeb4ee8e4b405184aee831bc69db185ae81ca8eea3a7785aa74fbe5a2b4df07"} Jul 28 17:58:19.545028 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:19.544974 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-chwbs"] Jul 28 17:58:19.547177 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:19.547146 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8495b74a_e4fa_4aee_a599_a46e993f7455.slice/crio-e664981233c062d586c25a1e3ef962016950a73fcef5ab55f58400bebb120548 WatchSource:0}: Error finding container e664981233c062d586c25a1e3ef962016950a73fcef5ab55f58400bebb120548: Status 404 returned error can't find the container with id e664981233c062d586c25a1e3ef962016950a73fcef5ab55f58400bebb120548 Jul 28 17:58:19.560101 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:19.560071 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj"] Jul 28 17:58:19.563156 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:19.563132 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1aa4c24_1149_41da_8bf8_e972ca19a17f.slice/crio-afc09f0b2a679da8eadc3a2c7b51671ba55b6ec433b483ceb1dd6db31c9a756b WatchSource:0}: Error finding container afc09f0b2a679da8eadc3a2c7b51671ba55b6ec433b483ceb1dd6db31c9a756b: Status 404 returned error can't find the container with id afc09f0b2a679da8eadc3a2c7b51671ba55b6ec433b483ceb1dd6db31c9a756b Jul 28 17:58:20.526836 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:20.526789 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" event={"ID":"b1aa4c24-1149-41da-8bf8-e972ca19a17f","Type":"ContainerStarted","Data":"afc09f0b2a679da8eadc3a2c7b51671ba55b6ec433b483ceb1dd6db31c9a756b"} Jul 28 17:58:20.528556 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:20.528525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-chwbs" event={"ID":"8495b74a-e4fa-4aee-a599-a46e993f7455","Type":"ContainerStarted","Data":"e664981233c062d586c25a1e3ef962016950a73fcef5ab55f58400bebb120548"} Jul 28 17:58:20.861899 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:20.861809 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:58:20.862075 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:20.861895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:58:20.869033 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:20.869004 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"image-registry-57cf8df6dc-bm9zl\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:58:20.869388 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:20.869364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0-metrics-tls\") pod \"dns-default-6ddcz\" (UID: \"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0\") " pod="openshift-dns/dns-default-6ddcz" Jul 28 17:58:21.117768 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.117286 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vrsc6\"" Jul 28 17:58:21.125287 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.125002 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:58:21.132957 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.132727 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5mqwj\"" Jul 28 17:58:21.141595 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.141533 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6ddcz" Jul 28 17:58:21.350255 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.350221 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-57cf8df6dc-bm9zl"] Jul 28 17:58:21.356757 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:21.356720 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29410269_90df_41a5_adcf_7d5c3136a321.slice/crio-d674d52906bf5f2f5b18d0b0903607fd220a95cc5d5ad8b6aec1477f2de48950 WatchSource:0}: Error finding container d674d52906bf5f2f5b18d0b0903607fd220a95cc5d5ad8b6aec1477f2de48950: Status 404 returned error can't find the container with id d674d52906bf5f2f5b18d0b0903607fd220a95cc5d5ad8b6aec1477f2de48950 Jul 28 17:58:21.376274 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.376016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6ddcz"] Jul 28 17:58:21.382187 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:21.382156 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0cb6d3_97f2_44b3_aaef_c5b1aa3ff1f0.slice/crio-7aa4c53ec76504fbf1f655c201c7d0d2b4f34ab4764efca1bdd1f87c72652543 WatchSource:0}: Error finding container 7aa4c53ec76504fbf1f655c201c7d0d2b4f34ab4764efca1bdd1f87c72652543: Status 404 returned error can't find the container with id 7aa4c53ec76504fbf1f655c201c7d0d2b4f34ab4764efca1bdd1f87c72652543 Jul 28 17:58:21.537441 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.537243 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ddcz" event={"ID":"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0","Type":"ContainerStarted","Data":"7aa4c53ec76504fbf1f655c201c7d0d2b4f34ab4764efca1bdd1f87c72652543"} Jul 28 17:58:21.543079 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.543043 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" event={"ID":"29410269-90df-41a5-adcf-7d5c3136a321","Type":"ContainerStarted","Data":"5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512"} Jul 28 17:58:21.543250 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.543126 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" event={"ID":"29410269-90df-41a5-adcf-7d5c3136a321","Type":"ContainerStarted","Data":"d674d52906bf5f2f5b18d0b0903607fd220a95cc5d5ad8b6aec1477f2de48950"} Jul 28 17:58:21.543917 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.543892 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:58:21.567918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:21.567212 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" podStartSLOduration=97.567188919 podStartE2EDuration="1m37.567188919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:21.56581865 +0000 UTC m=+98.980475666" watchObservedRunningTime="2026-07-28 17:58:21.567188919 +0000 UTC m=+98.981845923" Jul 28 17:58:24.073801 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:24.073764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:24.190730 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:24.190678 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:24.437036 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:24.437009 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-2qsvk" Jul 28 17:58:25.680154 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.680059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:25.781966 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.781914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:25.841732 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.841699 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746"] Jul 28 17:58:25.860361 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.860324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:25.865302 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.865270 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746"] Jul 28 17:58:25.865450 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.865421 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:25.868049 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.868022 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 28 17:58:25.868341 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:25.868325 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-rsc74\"" Jul 28 17:58:26.009334 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.009239 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b61a3559-21e8-4b9f-b2f3-efe1ed805953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9k746\" (UID: \"b61a3559-21e8-4b9f-b2f3-efe1ed805953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:26.058333 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.058276 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:26.109761 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.109722 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b61a3559-21e8-4b9f-b2f3-efe1ed805953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9k746\" (UID: \"b61a3559-21e8-4b9f-b2f3-efe1ed805953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:26.109969 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:26.109907 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 28 17:58:26.110041 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:26.110006 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b61a3559-21e8-4b9f-b2f3-efe1ed805953-tls-certificates podName:b61a3559-21e8-4b9f-b2f3-efe1ed805953 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:26.609984425 +0000 UTC m=+104.024641412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/b61a3559-21e8-4b9f-b2f3-efe1ed805953-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-9k746" (UID: "b61a3559-21e8-4b9f-b2f3-efe1ed805953") : secret "prometheus-operator-admission-webhook-tls" not found Jul 28 17:58:26.160703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.160601 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:26.222861 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.222824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:26.613764 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.613727 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b61a3559-21e8-4b9f-b2f3-efe1ed805953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9k746\" (UID: \"b61a3559-21e8-4b9f-b2f3-efe1ed805953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:26.617420 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.617385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b61a3559-21e8-4b9f-b2f3-efe1ed805953-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-9k746\" (UID: \"b61a3559-21e8-4b9f-b2f3-efe1ed805953\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:26.776805 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:26.776767 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:27.036918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.036840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:27.102390 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.102356 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:27.165800 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.165763 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:27.874703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.872815 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7787c98fd9-qn99d"] Jul 28 17:58:27.876039 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.876015 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.879926 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.878599 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 17:58:27.879926 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.878860 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-fg5x9\"" Jul 28 17:58:27.879926 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.879847 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 17:58:27.880175 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.880077 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 17:58:27.881406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.880273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 17:58:27.881406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.880451 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 17:58:27.890466 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.889244 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7787c98fd9-qn99d"] Jul 28 17:58:27.926709 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.926581 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-service-ca\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.926709 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.926630 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-config\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.926709 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.926658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-oauth-config\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.927762 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.926737 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/573e1cdc-5f0a-4078-b2d2-d25231e84224-kube-api-access-95tvd\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.927762 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.926856 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-serving-cert\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.927762 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.926894 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-oauth-serving-cert\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:27.979326 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:27.979219 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746"] Jul 28 17:58:27.987170 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:27.987132 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb61a3559_21e8_4b9f_b2f3_efe1ed805953.slice/crio-814c4c3948a250a01e53691c0fc4d37d8cafd5e6d1c4ce7f8a81c086e8bf2cd6 WatchSource:0}: Error finding container 814c4c3948a250a01e53691c0fc4d37d8cafd5e6d1c4ce7f8a81c086e8bf2cd6: Status 404 returned error can't find the container with id 814c4c3948a250a01e53691c0fc4d37d8cafd5e6d1c4ce7f8a81c086e8bf2cd6 Jul 28 17:58:28.027771 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.027738 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-serving-cert\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.027928 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.027780 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-oauth-serving-cert\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.027928 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.027826 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-service-ca\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.027928 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.027890 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-config\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.027928 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.027919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-oauth-config\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.028134 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.027944 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/573e1cdc-5f0a-4078-b2d2-d25231e84224-kube-api-access-95tvd\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.028891 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.028843 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-config\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.029008 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.028951 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-service-ca\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.029513 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.029483 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-oauth-serving-cert\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.031077 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.031056 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-oauth-config\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.031192 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.031174 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-serving-cert\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.037196 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.037167 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/573e1cdc-5f0a-4078-b2d2-d25231e84224-kube-api-access-95tvd\") pod \"console-7787c98fd9-qn99d\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.119543 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.119451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:28.188741 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.188702 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:28.217018 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.216979 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:28.370743 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.370645 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7787c98fd9-qn99d"] Jul 28 17:58:28.374405 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:28.374375 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573e1cdc_5f0a_4078_b2d2_d25231e84224.slice/crio-c3e6d9a6d3bcf774481bcf4ea4f2ff9b0298aa46e33cc1224288298bfa2893ca WatchSource:0}: Error finding container c3e6d9a6d3bcf774481bcf4ea4f2ff9b0298aa46e33cc1224288298bfa2893ca: Status 404 returned error can't find the container with id c3e6d9a6d3bcf774481bcf4ea4f2ff9b0298aa46e33cc1224288298bfa2893ca Jul 28 17:58:28.574359 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.574012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ddcz" event={"ID":"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0","Type":"ContainerStarted","Data":"0d3567cb3596ba950224634805e0bfb0653e3f20abf188aa36c109b1db17f596"} Jul 28 17:58:28.574359 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.574080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6ddcz" event={"ID":"8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0","Type":"ContainerStarted","Data":"ae0a2e4ffdc6adc310e08bd577765cd0613539a306d3d6595b24e50d4c87484e"} Jul 28 17:58:28.574800 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.574768 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6ddcz" Jul 28 17:58:28.577913 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.577876 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l9pkg" event={"ID":"51c0509b-69ea-4443-b397-6e7fb7313c19","Type":"ContainerStarted","Data":"ae8154d4a1eb13e7c75288b16733681766cab8252888f852a7232a9252c521d9"} Jul 28 17:58:28.580020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.579989 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" event={"ID":"b1aa4c24-1149-41da-8bf8-e972ca19a17f","Type":"ContainerStarted","Data":"47de04ae8a9664dfe73df5dd8eb2aa38955b4d059944adcda17d1a62249a1666"} Jul 28 17:58:28.580466 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.580448 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:28.582042 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.582017 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787c98fd9-qn99d" event={"ID":"573e1cdc-5f0a-4078-b2d2-d25231e84224","Type":"ContainerStarted","Data":"c3e6d9a6d3bcf774481bcf4ea4f2ff9b0298aa46e33cc1224288298bfa2893ca"} Jul 28 17:58:28.582279 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.582263 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" Jul 28 17:58:28.583729 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.583705 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" event={"ID":"b61a3559-21e8-4b9f-b2f3-efe1ed805953","Type":"ContainerStarted","Data":"814c4c3948a250a01e53691c0fc4d37d8cafd5e6d1c4ce7f8a81c086e8bf2cd6"} Jul 28 17:58:28.593435 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.593356 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6ddcz" podStartSLOduration=66.163176929 podStartE2EDuration="1m12.593337932s" totalImagesPullingTime="6.430161003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:57:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:28.591773998 +0000 UTC m=+106.006431001" watchObservedRunningTime="2026-07-28 17:58:28.593337932 +0000 UTC m=+106.007994936" Jul 28 17:58:28.628246 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.627765 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l9pkg" podStartSLOduration=17.13894671 podStartE2EDuration="33.627744429s" totalImagesPullingTime="16.488797719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:57:55 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:28.611091585 +0000 UTC m=+106.025748588" watchObservedRunningTime="2026-07-28 17:58:28.627744429 +0000 UTC m=+106.042401431" Jul 28 17:58:28.628527 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:28.628472 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7947cb57df-6bglj" podStartSLOduration=2.3563349430000002 podStartE2EDuration="10.628455981s" totalImagesPullingTime="8.272121038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:28.627233499 +0000 UTC m=+106.041904952" watchObservedRunningTime="2026-07-28 17:58:28.628455981 +0000 UTC m=+106.043112994" Jul 28 17:58:29.590617 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.590530 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" event={"ID":"b61a3559-21e8-4b9f-b2f3-efe1ed805953","Type":"ContainerStarted","Data":"dbeb3940ed5330f1dab4ab618ff8b108afb1c1db9adbb3e4b3dc8aaa2e769482"} Jul 28 17:58:29.591211 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.591136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:29.597302 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.597270 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" Jul 28 17:58:29.628887 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.628825 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-9k746" podStartSLOduration=3.460525386 podStartE2EDuration="4.628804498s" totalImagesPullingTime="1.168279112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:29.627768497 +0000 UTC m=+107.042425499" watchObservedRunningTime="2026-07-28 17:58:29.628804498 +0000 UTC m=+107.043461501" Jul 28 17:58:29.888504 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.888468 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:29.940215 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.939820 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kj4t7"] Jul 28 17:58:29.942364 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.942318 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:29.946544 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.946318 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 28 17:58:29.946673 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.946610 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 17:58:29.947580 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.947559 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 17:58:29.947783 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.947766 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 28 17:58:29.947901 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.947883 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 17:58:29.947957 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.947934 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-lplvp\"" Jul 28 17:58:29.965420 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:29.965331 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kj4t7"] Jul 28 17:58:30.038844 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.038759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:30.043913 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.043873 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbmz\" (UniqueName: \"kubernetes.io/projected/ecd696fc-76ef-4036-9cad-322083da49f9-kube-api-access-hvbmz\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.044067 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.043975 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.044067 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.044004 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd696fc-76ef-4036-9cad-322083da49f9-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.044177 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.044101 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.144849 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.144759 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.144849 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.144819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbmz\" (UniqueName: \"kubernetes.io/projected/ecd696fc-76ef-4036-9cad-322083da49f9-kube-api-access-hvbmz\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.145086 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.144885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.145086 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.144911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd696fc-76ef-4036-9cad-322083da49f9-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.145086 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:30.145030 2575 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 28 17:58:30.145236 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:30.145114 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-tls podName:ecd696fc-76ef-4036-9cad-322083da49f9 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:30.645093257 +0000 UTC m=+108.059750257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-kj4t7" (UID: "ecd696fc-76ef-4036-9cad-322083da49f9") : secret "prometheus-operator-tls" not found Jul 28 17:58:30.145982 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.145886 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ecd696fc-76ef-4036-9cad-322083da49f9-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.147818 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.147783 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.155341 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.155315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbmz\" (UniqueName: \"kubernetes.io/projected/ecd696fc-76ef-4036-9cad-322083da49f9-kube-api-access-hvbmz\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.249589 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.249505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:30.477231 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.477146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:30.592246 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.592211 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:30.650170 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.650042 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.652995 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.652968 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ecd696fc-76ef-4036-9cad-322083da49f9-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-kj4t7\" (UID: \"ecd696fc-76ef-4036-9cad-322083da49f9\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:30.854758 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:30.854662 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" Jul 28 17:58:31.258870 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:31.258834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:34.503231 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:34.503196 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:34.603835 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:34.603781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:35.866812 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:35.866775 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:35.969212 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:35.969178 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:38.667762 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:38.667732 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:38.727053 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:38.727022 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:39.293950 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.293921 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-kj4t7"] Jul 28 17:58:39.296909 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:39.296883 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecd696fc_76ef_4036_9cad_322083da49f9.slice/crio-c7ece1f6769d2a37705e7a0de04f1b4c57e01701d7e389911ca0b8001be09f0a WatchSource:0}: Error finding container c7ece1f6769d2a37705e7a0de04f1b4c57e01701d7e389911ca0b8001be09f0a: Status 404 returned error can't find the container with id c7ece1f6769d2a37705e7a0de04f1b4c57e01701d7e389911ca0b8001be09f0a Jul 28 17:58:39.591405 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.591306 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:39.597447 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.597419 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6ddcz" Jul 28 17:58:39.630787 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.630752 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787c98fd9-qn99d" event={"ID":"573e1cdc-5f0a-4078-b2d2-d25231e84224","Type":"ContainerStarted","Data":"650f4302681aef4c0bb0dcc70e24b44d74506321e48c52d4b9c697c5f0c4d619"} Jul 28 17:58:39.632100 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.632069 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" event={"ID":"ecd696fc-76ef-4036-9cad-322083da49f9","Type":"ContainerStarted","Data":"c7ece1f6769d2a37705e7a0de04f1b4c57e01701d7e389911ca0b8001be09f0a"} Jul 28 17:58:39.655372 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.655320 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7787c98fd9-qn99d" podStartSLOduration=1.8704352119999998 podStartE2EDuration="12.655305171s" totalImagesPullingTime="10.784869959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:39.653711959 +0000 UTC m=+117.068368964" watchObservedRunningTime="2026-07-28 17:58:39.655305171 +0000 UTC m=+117.069962172" Jul 28 17:58:39.700002 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.699968 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:39.861641 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:39.861547 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:40.642026 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:40.641929 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-chwbs" event={"ID":"8495b74a-e4fa-4aee-a599-a46e993f7455","Type":"ContainerStarted","Data":"42a7d7b932863e3b0af5c085f1ab8503ac59a2341241d378ec870570dcbb0b9c"} Jul 28 17:58:40.680777 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:40.680711 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-chwbs" podStartSLOduration=2.462593038 podStartE2EDuration="22.680693955s" totalImagesPullingTime="20.218100917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:40.679675859 +0000 UTC m=+118.094332863" watchObservedRunningTime="2026-07-28 17:58:40.680693955 +0000 UTC m=+118.095350947" Jul 28 17:58:41.383595 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.383549 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57cf8df6dc-bm9zl"] Jul 28 17:58:41.389334 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.389261 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:58:41.499928 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.499883 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:41.615646 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.615555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:41.648723 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.648657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" event={"ID":"ecd696fc-76ef-4036-9cad-322083da49f9","Type":"ContainerStarted","Data":"aeb1fba2c79bd4986712004f55e021903e7db7666ccf0fa13b564e958860bd7f"} Jul 28 17:58:41.648918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.648746 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" event={"ID":"ecd696fc-76ef-4036-9cad-322083da49f9","Type":"ContainerStarted","Data":"768c2cc79500648edec8db971076c1c5a225a43f02e405cd5ddb422e648bf108"} Jul 28 17:58:41.649155 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.649131 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:41.660410 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.660375 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-chwbs" Jul 28 17:58:41.680412 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.680343 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-kj4t7" podStartSLOduration=10.882583294 podStartE2EDuration="12.680320645s" totalImagesPullingTime="1.797737351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:41.678327049 +0000 UTC m=+119.092984051" watchObservedRunningTime="2026-07-28 17:58:41.680320645 +0000 UTC m=+119.094977648" Jul 28 17:58:41.762607 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:41.762569 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:43.096975 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.096933 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:43.264624 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.264577 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:43.391498 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.391459 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:43.424656 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.424612 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls"] Jul 28 17:58:43.465159 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.465118 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls"] Jul 28 17:58:43.465352 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.465180 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mbg9l"] Jul 28 17:58:43.465352 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.465274 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.470820 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.470789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 28 17:58:43.471106 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.471087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-l9td4\"" Jul 28 17:58:43.473880 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.473639 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 28 17:58:43.478881 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.478861 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.482988 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.482306 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 17:58:43.482988 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.482623 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-j6s8v\"" Jul 28 17:58:43.482988 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.482912 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 17:58:43.483372 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.483157 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 17:58:43.543832 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.543796 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566126 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-root\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566172 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-metrics-client-ca\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-wtmp\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phnp8\" (UniqueName: \"kubernetes.io/projected/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-kube-api-access-phnp8\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566254 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klq6k\" (UniqueName: \"kubernetes.io/projected/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-kube-api-access-klq6k\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566288 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566316 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-textfile\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566381 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566348 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-tls\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566431 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.566918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566468 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-sys\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.566918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.566918 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.566534 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667520 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667436 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-root\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667520 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-metrics-client-ca\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667520 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-wtmp\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667552 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phnp8\" (UniqueName: \"kubernetes.io/projected/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-kube-api-access-phnp8\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667579 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klq6k\" (UniqueName: \"kubernetes.io/projected/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-kube-api-access-klq6k\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667610 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667653 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-textfile\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667679 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-tls\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667732 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-sys\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.667863 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667834 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.668283 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.667879 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.668539 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.668515 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-accelerators-collector-config\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.668743 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.668714 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-root\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.668870 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:43.668852 2575 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 28 17:58:43.668935 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:43.668919 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-tls podName:e677163f-dbdd-48f9-b7a2-4a3b799d7fad nodeName:}" failed. No retries permitted until 2026-07-28 17:58:44.168902389 +0000 UTC m=+121.583559387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-tls") pod "node-exporter-mbg9l" (UID: "e677163f-dbdd-48f9-b7a2-4a3b799d7fad") : secret "node-exporter-tls" not found Jul 28 17:58:43.669192 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.669177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-textfile\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.669497 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.669473 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-metrics-client-ca\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.669755 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:43.669735 2575 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 28 17:58:43.669834 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:43.669802 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-tls podName:8bf3cdbc-00eb-48b7-81a1-936a4983dd1c nodeName:}" failed. No retries permitted until 2026-07-28 17:58:44.16978675 +0000 UTC m=+121.584443732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-8chls" (UID: "8bf3cdbc-00eb-48b7-81a1-936a4983dd1c") : secret "openshift-state-metrics-tls" not found Jul 28 17:58:43.669943 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.669914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-sys\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.670005 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.669984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-wtmp\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.670296 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.670278 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.672981 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.672649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.672981 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.672940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.686515 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.686445 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phnp8\" (UniqueName: \"kubernetes.io/projected/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-kube-api-access-phnp8\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:43.687373 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.687345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klq6k\" (UniqueName: \"kubernetes.io/projected/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-kube-api-access-klq6k\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:43.862424 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.862384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:43.951516 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:43.951420 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:44.129950 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.129893 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:44.173451 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.173137 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:44.173451 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.173199 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-tls\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:44.176431 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.176400 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/e677163f-dbdd-48f9-b7a2-4a3b799d7fad-node-exporter-tls\") pod \"node-exporter-mbg9l\" (UID: \"e677163f-dbdd-48f9-b7a2-4a3b799d7fad\") " pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:44.177513 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.177486 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8bf3cdbc-00eb-48b7-81a1-936a4983dd1c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8chls\" (UID: \"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:44.248237 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.248146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:44.353360 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.353317 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:44.379482 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.379445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" Jul 28 17:58:44.393698 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.393645 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mbg9l" Jul 28 17:58:44.456732 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:44.456656 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode677163f_dbdd_48f9_b7a2_4a3b799d7fad.slice/crio-b7891ec7fe89129368e0d2a9f719e60893cfc22493cc0c72ae740b88ba84db15 WatchSource:0}: Error finding container b7891ec7fe89129368e0d2a9f719e60893cfc22493cc0c72ae740b88ba84db15: Status 404 returned error can't find the container with id b7891ec7fe89129368e0d2a9f719e60893cfc22493cc0c72ae740b88ba84db15 Jul 28 17:58:44.474971 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.474936 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:44.504756 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.504368 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:58:44.544728 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.541303 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.544728 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.544070 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 17:58:44.547724 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.547431 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 17:58:44.547724 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.547449 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 17:58:44.547724 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.547579 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 17:58:44.547724 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.547712 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 17:58:44.548014 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.547773 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 17:58:44.548014 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.547954 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 17:58:44.548743 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.548177 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v9ggh\"" Jul 28 17:58:44.548743 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.548517 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 17:58:44.549753 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.549474 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 17:58:44.565048 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.565016 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:58:44.626696 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.626356 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls"] Jul 28 17:58:44.629321 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:44.629289 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf3cdbc_00eb_48b7_81a1_936a4983dd1c.slice/crio-8639e3fcc09968794115517adc0d2826f8055af98efbd54662fe0fe58c096f9d WatchSource:0}: Error finding container 8639e3fcc09968794115517adc0d2826f8055af98efbd54662fe0fe58c096f9d: Status 404 returned error can't find the container with id 8639e3fcc09968794115517adc0d2826f8055af98efbd54662fe0fe58c096f9d Jul 28 17:58:44.662866 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.662828 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbg9l" event={"ID":"e677163f-dbdd-48f9-b7a2-4a3b799d7fad","Type":"ContainerStarted","Data":"b7891ec7fe89129368e0d2a9f719e60893cfc22493cc0c72ae740b88ba84db15"} Jul 28 17:58:44.664309 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.664276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" event={"ID":"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c","Type":"ContainerStarted","Data":"8639e3fcc09968794115517adc0d2826f8055af98efbd54662fe0fe58c096f9d"} Jul 28 17:58:44.678665 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678631 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdxq\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-kube-api-access-spdxq\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.678850 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678700 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.678850 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678770 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.678850 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678851 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-web-config\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678878 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678948 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679020 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.678985 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679257 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.679042 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679257 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.679068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-out\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679257 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.679095 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.679257 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.679121 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780134 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780097 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780280 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-out\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780280 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780187 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780280 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780280 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780251 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdxq\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-kube-api-access-spdxq\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780280 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780277 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780330 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780374 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-web-config\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780439 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780466 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.780534 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.780508 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.781730 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.781676 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.781843 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:44.781818 2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 17:58:44.781910 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:44.781895 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls podName:ac6deb56-592f-4711-b557-d2eb7ffe2263 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:45.281872921 +0000 UTC m=+122.696529925 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263") : secret "alertmanager-main-tls" not found Jul 28 17:58:44.781996 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.781973 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.782926 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.782899 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-out\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.783357 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.783331 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.784356 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.784310 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-volume\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.786527 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.786490 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.787301 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.787228 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.788105 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.787307 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-web-config\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.788485 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.788409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.788837 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.788809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.789551 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.789534 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.803241 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.802977 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdxq\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-kube-api-access-spdxq\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:44.934412 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:44.934337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:45.031348 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.031179 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:45.286430 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.286346 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:45.286890 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:45.286526 2575 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 28 17:58:45.286890 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:58:45.286608 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls podName:ac6deb56-592f-4711-b557-d2eb7ffe2263 nodeName:}" failed. No retries permitted until 2026-07-28 17:58:46.286586334 +0000 UTC m=+123.701243314 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263") : secret "alertmanager-main-tls" not found Jul 28 17:58:45.519044 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.519006 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v"] Jul 28 17:58:45.544698 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.544604 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7787c98fd9-qn99d"] Jul 28 17:58:45.544698 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.544663 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v"] Jul 28 17:58:45.545515 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.545106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.549410 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7ioha3qvfoldf\"" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.549495 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.549565 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.549423 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.549748 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.549789 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 28 17:58:45.550312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.550050 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6htf9\"" Jul 28 17:58:45.589587 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589544 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.589791 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74c783e0-b4db-4820-a21a-f81a829763bd-metrics-client-ca\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.589791 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589704 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.589903 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589790 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-grpc-tls\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.589903 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589823 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.589903 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589859 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8rk\" (UniqueName: \"kubernetes.io/projected/74c783e0-b4db-4820-a21a-f81a829763bd-kube-api-access-lp8rk\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.590049 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589927 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-tls\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.590049 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.589989 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.602796 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.602764 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:45.671159 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.671119 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" event={"ID":"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c","Type":"ContainerStarted","Data":"30169a3e86d24bb91ac6c5e1957fb66c3d243b4728763f34501cf3b88ad40ef3"} Jul 28 17:58:45.671357 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.671200 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" event={"ID":"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c","Type":"ContainerStarted","Data":"a7b446158bcffa6617101bf514733c2b98c23b8075feee54442976e335a31772"} Jul 28 17:58:45.689162 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.689125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:45.691244 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691190 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74c783e0-b4db-4820-a21a-f81a829763bd-metrics-client-ca\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691280 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691323 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-grpc-tls\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691406 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691362 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691579 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691414 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8rk\" (UniqueName: \"kubernetes.io/projected/74c783e0-b4db-4820-a21a-f81a829763bd-kube-api-access-lp8rk\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691579 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691442 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-tls\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691841 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.691924 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.691874 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.692842 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.692813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/74c783e0-b4db-4820-a21a-f81a829763bd-metrics-client-ca\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.694720 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.694618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.694873 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.694850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.694949 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.694926 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.695369 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.695347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-grpc-tls\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.695925 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.695903 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-tls\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.696284 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.696256 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/74c783e0-b4db-4820-a21a-f81a829763bd-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.706333 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.706300 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8rk\" (UniqueName: \"kubernetes.io/projected/74c783e0-b4db-4820-a21a-f81a829763bd-kube-api-access-lp8rk\") pod \"thanos-querier-c4cdbfd9b-pgv2v\" (UID: \"74c783e0-b4db-4820-a21a-f81a829763bd\") " pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:45.790191 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.790149 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:45.857938 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.857850 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:45.858663 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:45.858632 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:46.090806 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.090665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:46.139386 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.139349 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v"] Jul 28 17:58:46.142233 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:46.142191 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c783e0_b4db_4820_a21a_f81a829763bd.slice/crio-a9706ed2785a2293c87afb3391ab51402304c967e59c654513604b1d1f14a029 WatchSource:0}: Error finding container a9706ed2785a2293c87afb3391ab51402304c967e59c654513604b1d1f14a029: Status 404 returned error can't find the container with id a9706ed2785a2293c87afb3391ab51402304c967e59c654513604b1d1f14a029 Jul 28 17:58:46.186227 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.186157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:46.299135 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.299093 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:46.302239 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.302205 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:46.358657 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.358615 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 17:58:46.391066 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.390984 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:46.457329 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.457289 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:46.679378 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.679282 2575 generic.go:356] "Generic (PLEG): container finished" podID="e677163f-dbdd-48f9-b7a2-4a3b799d7fad" containerID="34197b9bdfca9d66ead1627bac06e6c94c7d5e5f5f5cdf4f2bc019b421186186" exitCode=0 Jul 28 17:58:46.679657 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.679367 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbg9l" event={"ID":"e677163f-dbdd-48f9-b7a2-4a3b799d7fad","Type":"ContainerDied","Data":"34197b9bdfca9d66ead1627bac06e6c94c7d5e5f5f5cdf4f2bc019b421186186"} Jul 28 17:58:46.681455 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.681428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"a9706ed2785a2293c87afb3391ab51402304c967e59c654513604b1d1f14a029"} Jul 28 17:58:46.702561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.702524 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:46.810201 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:46.810134 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:47.018983 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.018933 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 17:58:47.187302 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:47.187211 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6deb56_592f_4711_b557_d2eb7ffe2263.slice/crio-cb33b5043de94393f6e5bacf29f24db98ef3d9bc92b614cdf3defa8d26857d79 WatchSource:0}: Error finding container cb33b5043de94393f6e5bacf29f24db98ef3d9bc92b614cdf3defa8d26857d79: Status 404 returned error can't find the container with id cb33b5043de94393f6e5bacf29f24db98ef3d9bc92b614cdf3defa8d26857d79 Jul 28 17:58:47.428175 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.428136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:47.529388 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.529303 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:47.688052 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.688005 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"cb33b5043de94393f6e5bacf29f24db98ef3d9bc92b614cdf3defa8d26857d79"} Jul 28 17:58:47.690965 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.690918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbg9l" event={"ID":"e677163f-dbdd-48f9-b7a2-4a3b799d7fad","Type":"ContainerStarted","Data":"09af38f819ca433afa41425b0024c2c072e4d075f3fb96ad65ef66ec6e0abf4a"} Jul 28 17:58:47.691129 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.690982 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mbg9l" event={"ID":"e677163f-dbdd-48f9-b7a2-4a3b799d7fad","Type":"ContainerStarted","Data":"6a3eb9bef813a423d9f2960939ca96da3a7a7c4bac3e351526813658eb7b3423"} Jul 28 17:58:47.694427 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.694390 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" event={"ID":"8bf3cdbc-00eb-48b7-81a1-936a4983dd1c","Type":"ContainerStarted","Data":"5d65e0bd5914d8e993050aba19297e13baeca0de28f74ed30da328b395f42840"} Jul 28 17:58:47.712661 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.712595 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mbg9l" podStartSLOduration=3.11762213 podStartE2EDuration="4.712571912s" totalImagesPullingTime="1.496131034s" totalInitContainerRuntime="98.818748ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:47.710878727 +0000 UTC m=+125.125535730" watchObservedRunningTime="2026-07-28 17:58:47.712571912 +0000 UTC m=+125.127228916" Jul 28 17:58:47.732425 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:47.732362 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8chls" podStartSLOduration=2.460817272 podStartE2EDuration="4.732340672s" totalImagesPullingTime="2.2715234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:47.731032541 +0000 UTC m=+125.145689543" watchObservedRunningTime="2026-07-28 17:58:47.732340672 +0000 UTC m=+125.146997674" Jul 28 17:58:48.093076 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.093027 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:48.188964 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.188921 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:48.217633 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.217548 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:58:48.811630 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.811582 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56689ffbcc-knn76"] Jul 28 17:58:48.834731 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.833293 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56689ffbcc-knn76"] Jul 28 17:58:48.834731 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.833411 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.850303 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.850273 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 17:58:48.911408 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.911359 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:48.926490 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9qrd\" (UniqueName: \"kubernetes.io/projected/f03f4d1a-f727-4bf1-960a-086b85c90445-kube-api-access-k9qrd\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.926703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-service-ca\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.926703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926569 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-trusted-ca-bundle\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.926703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-serving-cert\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.926703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926661 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-oauth-serving-cert\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.926703 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926697 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-oauth-config\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:48.926985 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:48.926743 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-console-config\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028044 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028007 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9qrd\" (UniqueName: \"kubernetes.io/projected/f03f4d1a-f727-4bf1-960a-086b85c90445-kube-api-access-k9qrd\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028209 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028098 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-service-ca\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028209 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028121 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-trusted-ca-bundle\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028209 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028149 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-serving-cert\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028209 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-oauth-serving-cert\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028209 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028196 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-oauth-config\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028421 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028225 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-console-config\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.028974 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.028943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-service-ca\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.029090 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.029040 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-console-config\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.029159 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.029109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-oauth-serving-cert\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.029821 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.029795 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-trusted-ca-bundle\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.031041 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.031008 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-oauth-config\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.031297 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.031276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-serving-cert\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.034094 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.034066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:49.036850 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.036824 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9qrd\" (UniqueName: \"kubernetes.io/projected/f03f4d1a-f727-4bf1-960a-086b85c90445-kube-api-access-k9qrd\") pod \"console-56689ffbcc-knn76\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.150486 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.150445 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:49.684342 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.684295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:49.866744 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:49.866460 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56689ffbcc-knn76"] Jul 28 17:58:49.873353 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:49.872206 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03f4d1a_f727_4bf1_960a_086b85c90445.slice/crio-20334e1420ea6a0894a0e1213b9845f8999bbd273aff2e92cd4de75a10e19455 WatchSource:0}: Error finding container 20334e1420ea6a0894a0e1213b9845f8999bbd273aff2e92cd4de75a10e19455: Status 404 returned error can't find the container with id 20334e1420ea6a0894a0e1213b9845f8999bbd273aff2e92cd4de75a10e19455 Jul 28 17:58:50.710824 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.710783 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="baad2cab860beb3dacae9b70eaa549be195670a77303922538a70d0cdeb519ec" exitCode=0 Jul 28 17:58:50.711038 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.710877 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"baad2cab860beb3dacae9b70eaa549be195670a77303922538a70d0cdeb519ec"} Jul 28 17:58:50.712579 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.712550 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56689ffbcc-knn76" event={"ID":"f03f4d1a-f727-4bf1-960a-086b85c90445","Type":"ContainerStarted","Data":"c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988"} Jul 28 17:58:50.712731 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.712603 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56689ffbcc-knn76" event={"ID":"f03f4d1a-f727-4bf1-960a-086b85c90445","Type":"ContainerStarted","Data":"20334e1420ea6a0894a0e1213b9845f8999bbd273aff2e92cd4de75a10e19455"} Jul 28 17:58:50.714698 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.714657 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"13bfc6969d0671d77ba269a91b61b6e1507b5b62d47f4519db3bf85f0cdaa8cc"} Jul 28 17:58:50.714783 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.714735 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"e826919a9d781930c8ae1b79fa388292884ee9d20c4bd3c527a6679c02268307"} Jul 28 17:58:50.714783 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.714763 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"b6ec363b67c0caae5f72a8b02d8ad7c18b01ce0f31558cb54b51498145505729"} Jul 28 17:58:50.761603 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:50.761534 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56689ffbcc-knn76" podStartSLOduration=2.761514311 podStartE2EDuration="2.761514311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:50.760004622 +0000 UTC m=+128.174661625" watchObservedRunningTime="2026-07-28 17:58:50.761514311 +0000 UTC m=+128.176171314" Jul 28 17:58:51.722224 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:51.722190 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"df7c515d5c1373dff92cd06dbaac5ed2923fc61b95ccd36c9b59697681084f1f"} Jul 28 17:58:51.722612 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:51.722263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"af27f971f4186861d3a77d9b9d3982b22aba67b345486a0476173de226462506"} Jul 28 17:58:51.860728 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:51.858380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:58:51.864880 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:51.864850 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/144d01ab-98a5-4f4c-bee9-a4030a422bec-metrics-certs\") pod \"network-metrics-daemon-fjvtt\" (UID: \"144d01ab-98a5-4f4c-bee9-a4030a422bec\") " pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:58:51.881656 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:51.881625 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pbl4v\"" Jul 28 17:58:51.888955 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:51.888930 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fjvtt" Jul 28 17:58:52.045891 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.045866 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fjvtt"] Jul 28 17:58:52.047829 ip-10-0-135-147 kubenswrapper[2575]: W0728 17:58:52.047797 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144d01ab_98a5_4f4c_bee9_a4030a422bec.slice/crio-5108045b82f149c444a13ab42be68f9c3bf60c505e29ac44298e448b55a82950 WatchSource:0}: Error finding container 5108045b82f149c444a13ab42be68f9c3bf60c505e29ac44298e448b55a82950: Status 404 returned error can't find the container with id 5108045b82f149c444a13ab42be68f9c3bf60c505e29ac44298e448b55a82950 Jul 28 17:58:52.130308 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.130277 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:52.196839 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.196793 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:52.730515 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.730474 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjvtt" event={"ID":"144d01ab-98a5-4f4c-bee9-a4030a422bec","Type":"ContainerStarted","Data":"5108045b82f149c444a13ab42be68f9c3bf60c505e29ac44298e448b55a82950"} Jul 28 17:58:52.734301 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.734258 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" event={"ID":"74c783e0-b4db-4820-a21a-f81a829763bd","Type":"ContainerStarted","Data":"06a1f889d7887c38e08137b11757e0c67b36f2cd5d2ada89a861567a1df9251d"} Jul 28 17:58:52.734481 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.734409 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:52.760860 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:52.760802 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" podStartSLOduration=2.871557632 podStartE2EDuration="7.760782713s" totalImagesPullingTime="4.889225081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:45 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:52.758863952 +0000 UTC m=+130.173520956" watchObservedRunningTime="2026-07-28 17:58:52.760782713 +0000 UTC m=+130.175439714" Jul 28 17:58:54.749616 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.749583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"92e26582e595be6fdd9ea49e0d47ea77867795f5de15528b2bc13e24063aca92"} Jul 28 17:58:54.750117 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.749664 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"775b8461dc319fc426bb4e1ad5d5e5c7e576e55731fe75dfa39299406484a69f"} Jul 28 17:58:54.750117 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.749730 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"3c57d4d8edb4cc3305bb83a25f45617318a231c4d107a4cd5dfd028096beb121"} Jul 28 17:58:54.750117 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.749780 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"a0043f7d029d944968e15aacd277a738b8916693a8bdcf5e9986d5a53d69000d"} Jul 28 17:58:54.751337 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.751309 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjvtt" event={"ID":"144d01ab-98a5-4f4c-bee9-a4030a422bec","Type":"ContainerStarted","Data":"f4fe6fa317ed627eae89f0b44fd0e939028cd636ffdadbfd9a294196445ad546"} Jul 28 17:58:54.751447 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.751358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fjvtt" event={"ID":"144d01ab-98a5-4f4c-bee9-a4030a422bec","Type":"ContainerStarted","Data":"2c40995b79d0d86b33bf5a08a58ea9b03c3d9ef60c0216d360d9ea40ba9070ec"} Jul 28 17:58:54.779507 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.779447 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fjvtt" podStartSLOduration=129.919069398 podStartE2EDuration="2m11.779298735s" totalImagesPullingTime="1.860229337s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:56:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:54.777328347 +0000 UTC m=+132.191985350" watchObservedRunningTime="2026-07-28 17:58:54.779298735 +0000 UTC m=+132.193955736" Jul 28 17:58:54.954416 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:54.954378 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:55.049778 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:55.049662 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:55.157119 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:55.157084 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:58:55.759097 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:55.759056 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"c92a7765ee2bff9b87e5157d7daa43988a725d96eb77bc3dd4ab55c15889441b"} Jul 28 17:58:55.759574 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:55.759129 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerStarted","Data":"17d1804f4ffabdc29a99e452376ddba9dade8eec1a571e471caedcedd8050c4d"} Jul 28 17:58:55.788568 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:55.788460 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.750200263 podStartE2EDuration="11.788444994s" totalImagesPullingTime="6.019411114s" totalInitContainerRuntime="18.833617ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:58:44 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:58:55.787302558 +0000 UTC m=+133.201959561" watchObservedRunningTime="2026-07-28 17:58:55.788444994 +0000 UTC m=+133.203101996" Jul 28 17:58:58.745480 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:58.745450 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-c4cdbfd9b-pgv2v" Jul 28 17:58:59.151271 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:59.151233 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:59.151449 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:59.151298 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:59.155888 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:59.155861 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:59.780983 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:59.780947 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 17:58:59.960561 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:58:59.960522 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:00.060650 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:00.060559 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:00.160171 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:00.160105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:03.117612 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:03.117571 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:03.218280 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:03.218248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:06.409531 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.409456 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" podUID="29410269-90df-41a5-adcf-7d5c3136a321" containerName="registry" containerID="cri-o://5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512" gracePeriod=30 Jul 28 17:59:06.693310 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.693229 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:59:06.794104 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794069 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-registry-certificates\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794276 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794126 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794276 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794144 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-trusted-ca\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794276 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794178 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-installation-pull-secrets\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794276 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794195 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rggm\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-kube-api-access-2rggm\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794276 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794227 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-image-registry-private-configuration\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794521 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794366 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29410269-90df-41a5-adcf-7d5c3136a321-ca-trust-extracted\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794521 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794418 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-bound-sa-token\") pod \"29410269-90df-41a5-adcf-7d5c3136a321\" (UID: \"29410269-90df-41a5-adcf-7d5c3136a321\") " Jul 28 17:59:06.794611 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794585 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:06.794743 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794680 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:06.794999 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.794778 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-trusted-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.796785 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.796756 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:06.796923 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.796804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-kube-api-access-2rggm" (OuterVolumeSpecName: "kube-api-access-2rggm") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "kube-api-access-2rggm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:06.796985 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.796933 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:59:06.797067 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.797035 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:59:06.797118 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.797049 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:06.806194 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.806161 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29410269-90df-41a5-adcf-7d5c3136a321-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "29410269-90df-41a5-adcf-7d5c3136a321" (UID: "29410269-90df-41a5-adcf-7d5c3136a321"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 17:59:06.807158 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.807134 2575 generic.go:356] "Generic (PLEG): container finished" podID="29410269-90df-41a5-adcf-7d5c3136a321" containerID="5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512" exitCode=0 Jul 28 17:59:06.807241 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.807194 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" Jul 28 17:59:06.807241 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.807213 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" event={"ID":"29410269-90df-41a5-adcf-7d5c3136a321","Type":"ContainerDied","Data":"5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512"} Jul 28 17:59:06.807312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.807263 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-57cf8df6dc-bm9zl" event={"ID":"29410269-90df-41a5-adcf-7d5c3136a321","Type":"ContainerDied","Data":"d674d52906bf5f2f5b18d0b0903607fd220a95cc5d5ad8b6aec1477f2de48950"} Jul 28 17:59:06.807312 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.807294 2575 scope.go:117] "RemoveContainer" containerID="5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512" Jul 28 17:59:06.818712 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.818676 2575 scope.go:117] "RemoveContainer" containerID="5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512" Jul 28 17:59:06.818965 ip-10-0-135-147 kubenswrapper[2575]: E0728 17:59:06.818948 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512\": container with ID starting with 5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512 not found: ID does not exist" containerID="5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512" Jul 28 17:59:06.819019 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.818974 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512"} err="failed to get container status \"5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512\": rpc error: code = NotFound desc = could not find container \"5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512\": container with ID starting with 5847e0ab246c1946d3ad47b926e81595286c93ac15e54154a9715b48a38af512 not found: ID does not exist" Jul 28 17:59:06.831796 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.831772 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-57cf8df6dc-bm9zl"] Jul 28 17:59:06.835472 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.835449 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-57cf8df6dc-bm9zl"] Jul 28 17:59:06.895864 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895823 2575 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/29410269-90df-41a5-adcf-7d5c3136a321-registry-certificates\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.895864 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895860 2575 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-registry-tls\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.895864 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895873 2575 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-installation-pull-secrets\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.896068 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895882 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rggm\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-kube-api-access-2rggm\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.896068 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895892 2575 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/29410269-90df-41a5-adcf-7d5c3136a321-image-registry-private-configuration\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.896068 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895901 2575 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/29410269-90df-41a5-adcf-7d5c3136a321-ca-trust-extracted\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:06.896068 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:06.895911 2575 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/29410269-90df-41a5-adcf-7d5c3136a321-bound-sa-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:07.058294 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:07.058220 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29410269-90df-41a5-adcf-7d5c3136a321" path="/var/lib/kubelet/pods/29410269-90df-41a5-adcf-7d5c3136a321/volumes" Jul 28 17:59:10.577647 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.577600 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7787c98fd9-qn99d" podUID="573e1cdc-5f0a-4078-b2d2-d25231e84224" containerName="console" containerID="cri-o://650f4302681aef4c0bb0dcc70e24b44d74506321e48c52d4b9c697c5f0c4d619" gracePeriod=15 Jul 28 17:59:10.830257 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.830223 2575 generic.go:356] "Generic (PLEG): container finished" podID="573e1cdc-5f0a-4078-b2d2-d25231e84224" containerID="650f4302681aef4c0bb0dcc70e24b44d74506321e48c52d4b9c697c5f0c4d619" exitCode=0 Jul 28 17:59:10.830390 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.830267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787c98fd9-qn99d" event={"ID":"573e1cdc-5f0a-4078-b2d2-d25231e84224","Type":"ContainerDied","Data":"650f4302681aef4c0bb0dcc70e24b44d74506321e48c52d4b9c697c5f0c4d619"} Jul 28 17:59:10.830390 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.830337 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7787c98fd9-qn99d" event={"ID":"573e1cdc-5f0a-4078-b2d2-d25231e84224","Type":"ContainerDied","Data":"c3e6d9a6d3bcf774481bcf4ea4f2ff9b0298aa46e33cc1224288298bfa2893ca"} Jul 28 17:59:10.830390 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.830360 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3e6d9a6d3bcf774481bcf4ea4f2ff9b0298aa46e33cc1224288298bfa2893ca" Jul 28 17:59:10.864842 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.864805 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:59:10.929696 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.929662 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-oauth-config\") pod \"573e1cdc-5f0a-4078-b2d2-d25231e84224\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " Jul 28 17:59:10.929879 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.929705 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-serving-cert\") pod \"573e1cdc-5f0a-4078-b2d2-d25231e84224\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " Jul 28 17:59:10.929879 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.929742 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-oauth-serving-cert\") pod \"573e1cdc-5f0a-4078-b2d2-d25231e84224\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " Jul 28 17:59:10.929879 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.929773 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-config\") pod \"573e1cdc-5f0a-4078-b2d2-d25231e84224\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " Jul 28 17:59:10.929879 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.929816 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-service-ca\") pod \"573e1cdc-5f0a-4078-b2d2-d25231e84224\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " Jul 28 17:59:10.930240 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.930201 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "573e1cdc-5f0a-4078-b2d2-d25231e84224" (UID: "573e1cdc-5f0a-4078-b2d2-d25231e84224"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:10.930240 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.930229 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-config" (OuterVolumeSpecName: "console-config") pod "573e1cdc-5f0a-4078-b2d2-d25231e84224" (UID: "573e1cdc-5f0a-4078-b2d2-d25231e84224"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:10.930382 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.930271 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-service-ca" (OuterVolumeSpecName: "service-ca") pod "573e1cdc-5f0a-4078-b2d2-d25231e84224" (UID: "573e1cdc-5f0a-4078-b2d2-d25231e84224"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 17:59:10.932093 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.932070 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "573e1cdc-5f0a-4078-b2d2-d25231e84224" (UID: "573e1cdc-5f0a-4078-b2d2-d25231e84224"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:59:10.932200 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:10.932123 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "573e1cdc-5f0a-4078-b2d2-d25231e84224" (UID: "573e1cdc-5f0a-4078-b2d2-d25231e84224"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 17:59:11.030575 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.030534 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/573e1cdc-5f0a-4078-b2d2-d25231e84224-kube-api-access-95tvd\") pod \"573e1cdc-5f0a-4078-b2d2-d25231e84224\" (UID: \"573e1cdc-5f0a-4078-b2d2-d25231e84224\") " Jul 28 17:59:11.030757 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.030710 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:11.030757 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.030726 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:11.030757 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.030736 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-oauth-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:11.030757 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.030746 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/573e1cdc-5f0a-4078-b2d2-d25231e84224-console-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:11.030757 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.030755 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/573e1cdc-5f0a-4078-b2d2-d25231e84224-oauth-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:11.032551 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.032525 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573e1cdc-5f0a-4078-b2d2-d25231e84224-kube-api-access-95tvd" (OuterVolumeSpecName: "kube-api-access-95tvd") pod "573e1cdc-5f0a-4078-b2d2-d25231e84224" (UID: "573e1cdc-5f0a-4078-b2d2-d25231e84224"). InnerVolumeSpecName "kube-api-access-95tvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 17:59:11.131315 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.131281 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95tvd\" (UniqueName: \"kubernetes.io/projected/573e1cdc-5f0a-4078-b2d2-d25231e84224-kube-api-access-95tvd\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 17:59:11.833885 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.833852 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7787c98fd9-qn99d" Jul 28 17:59:11.852132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.852098 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7787c98fd9-qn99d"] Jul 28 17:59:11.856821 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:11.856792 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7787c98fd9-qn99d"] Jul 28 17:59:12.677195 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:12.677147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:12.778083 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:12.778048 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:13.063126 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:13.063035 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="573e1cdc-5f0a-4078-b2d2-d25231e84224" path="/var/lib/kubelet/pods/573e1cdc-5f0a-4078-b2d2-d25231e84224/volumes" Jul 28 17:59:15.205132 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:15.205105 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kwltp_b5cfce10-c28f-463f-ab5e-0626d77afe0d/serve-healthcheck-canary/0.log" Jul 28 17:59:17.333496 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:17.333461 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:18.113526 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:18.113451 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:18.178286 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:18.178249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:25.220239 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:25.220194 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:25.322113 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:25.322077 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 17:59:25.387585 ip-10-0-135-147 kubenswrapper[2575]: I0728 17:59:25.387543 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:02.715575 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:02.715538 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:03.115008 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.114890 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:03.819800 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.819759 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 18:00:03.820202 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.820098 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="alertmanager" containerID="cri-o://a0043f7d029d944968e15aacd277a738b8916693a8bdcf5e9986d5a53d69000d" gracePeriod=120 Jul 28 18:00:03.820269 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.820168 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-metric" containerID="cri-o://17d1804f4ffabdc29a99e452376ddba9dade8eec1a571e471caedcedd8050c4d" gracePeriod=120 Jul 28 18:00:03.820269 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.820195 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-web" containerID="cri-o://775b8461dc319fc426bb4e1ad5d5e5c7e576e55731fe75dfa39299406484a69f" gracePeriod=120 Jul 28 18:00:03.820269 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.820219 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy" containerID="cri-o://92e26582e595be6fdd9ea49e0d47ea77867795f5de15528b2bc13e24063aca92" gracePeriod=120 Jul 28 18:00:03.820419 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.820240 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="config-reloader" containerID="cri-o://3c57d4d8edb4cc3305bb83a25f45617318a231c4d107a4cd5dfd028096beb121" gracePeriod=120 Jul 28 18:00:03.820419 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:03.820326 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="prom-label-proxy" containerID="cri-o://c92a7765ee2bff9b87e5157d7daa43988a725d96eb77bc3dd4ab55c15889441b" gracePeriod=120 Jul 28 18:00:04.041396 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041363 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="c92a7765ee2bff9b87e5157d7daa43988a725d96eb77bc3dd4ab55c15889441b" exitCode=0 Jul 28 18:00:04.041396 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041388 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="92e26582e595be6fdd9ea49e0d47ea77867795f5de15528b2bc13e24063aca92" exitCode=0 Jul 28 18:00:04.041396 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041395 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="3c57d4d8edb4cc3305bb83a25f45617318a231c4d107a4cd5dfd028096beb121" exitCode=0 Jul 28 18:00:04.041396 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041400 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="a0043f7d029d944968e15aacd277a738b8916693a8bdcf5e9986d5a53d69000d" exitCode=0 Jul 28 18:00:04.041653 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"c92a7765ee2bff9b87e5157d7daa43988a725d96eb77bc3dd4ab55c15889441b"} Jul 28 18:00:04.041653 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"92e26582e595be6fdd9ea49e0d47ea77867795f5de15528b2bc13e24063aca92"} Jul 28 18:00:04.041653 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"3c57d4d8edb4cc3305bb83a25f45617318a231c4d107a4cd5dfd028096beb121"} Jul 28 18:00:04.041653 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:04.041548 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"a0043f7d029d944968e15aacd277a738b8916693a8bdcf5e9986d5a53d69000d"} Jul 28 18:00:05.049035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.049004 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="17d1804f4ffabdc29a99e452376ddba9dade8eec1a571e471caedcedd8050c4d" exitCode=0 Jul 28 18:00:05.049035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.049027 2575 generic.go:356] "Generic (PLEG): container finished" podID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerID="775b8461dc319fc426bb4e1ad5d5e5c7e576e55731fe75dfa39299406484a69f" exitCode=0 Jul 28 18:00:05.049571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.049082 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"17d1804f4ffabdc29a99e452376ddba9dade8eec1a571e471caedcedd8050c4d"} Jul 28 18:00:05.049571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.049151 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"775b8461dc319fc426bb4e1ad5d5e5c7e576e55731fe75dfa39299406484a69f"} Jul 28 18:00:05.075075 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.075048 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:05.081594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081565 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-metrics-client-ca\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081709 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081608 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-web-config\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081709 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081635 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-volume\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081709 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081696 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spdxq\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-kube-api-access-spdxq\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081889 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081815 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-out\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081889 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081866 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-main-db\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081890 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081916 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-tls-assets\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081946 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-cluster-tls-config\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.081991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081976 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-trusted-ca-bundle\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.082195 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.081985 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:05.082195 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082006 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.082195 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082053 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-web\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.082195 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082098 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ac6deb56-592f-4711-b557-d2eb7ffe2263\" (UID: \"ac6deb56-592f-4711-b557-d2eb7ffe2263\") " Jul 28 18:00:05.082399 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082229 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:05.082399 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082354 2575 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-metrics-client-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.082399 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082372 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-main-db\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.083033 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.082999 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:05.084754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.084650 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:00:05.085152 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.085124 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-kube-api-access-spdxq" (OuterVolumeSpecName: "kube-api-access-spdxq") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "kube-api-access-spdxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:00:05.085325 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.085293 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.085436 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.085327 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-volume" (OuterVolumeSpecName: "config-volume") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.085603 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.085577 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.085809 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.085786 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.085921 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.085871 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.086270 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.086248 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-out" (OuterVolumeSpecName: "config-out") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:00:05.089638 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.089595 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.096389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.096358 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-web-config" (OuterVolumeSpecName: "web-config") pod "ac6deb56-592f-4711-b557-d2eb7ffe2263" (UID: "ac6deb56-592f-4711-b557-d2eb7ffe2263"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:05.183322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183279 2575 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac6deb56-592f-4711-b557-d2eb7ffe2263-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183320 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183337 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183350 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183366 2575 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-web-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183377 2575 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-volume\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183387 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdxq\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-kube-api-access-spdxq\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183400 2575 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ac6deb56-592f-4711-b557-d2eb7ffe2263-config-out\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183411 2575 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-secret-alertmanager-main-tls\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183423 2575 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ac6deb56-592f-4711-b557-d2eb7ffe2263-tls-assets\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:05.183569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:05.183434 2575 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ac6deb56-592f-4711-b557-d2eb7ffe2263-cluster-tls-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:06.056260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.056223 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ac6deb56-592f-4711-b557-d2eb7ffe2263","Type":"ContainerDied","Data":"cb33b5043de94393f6e5bacf29f24db98ef3d9bc92b614cdf3defa8d26857d79"} Jul 28 18:00:06.056668 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.056284 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.056668 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.056297 2575 scope.go:117] "RemoveContainer" containerID="c92a7765ee2bff9b87e5157d7daa43988a725d96eb77bc3dd4ab55c15889441b" Jul 28 18:00:06.063939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.063920 2575 scope.go:117] "RemoveContainer" containerID="17d1804f4ffabdc29a99e452376ddba9dade8eec1a571e471caedcedd8050c4d" Jul 28 18:00:06.071730 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.071711 2575 scope.go:117] "RemoveContainer" containerID="92e26582e595be6fdd9ea49e0d47ea77867795f5de15528b2bc13e24063aca92" Jul 28 18:00:06.079179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.079157 2575 scope.go:117] "RemoveContainer" containerID="775b8461dc319fc426bb4e1ad5d5e5c7e576e55731fe75dfa39299406484a69f" Jul 28 18:00:06.084449 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.084422 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 18:00:06.087248 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.087228 2575 scope.go:117] "RemoveContainer" containerID="3c57d4d8edb4cc3305bb83a25f45617318a231c4d107a4cd5dfd028096beb121" Jul 28 18:00:06.089648 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.089624 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 18:00:06.095363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.095344 2575 scope.go:117] "RemoveContainer" containerID="a0043f7d029d944968e15aacd277a738b8916693a8bdcf5e9986d5a53d69000d" Jul 28 18:00:06.103057 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.103040 2575 scope.go:117] "RemoveContainer" containerID="baad2cab860beb3dacae9b70eaa549be195670a77303922538a70d0cdeb519ec" Jul 28 18:00:06.121561 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121524 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 18:00:06.121873 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121834 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="29410269-90df-41a5-adcf-7d5c3136a321" containerName="registry" Jul 28 18:00:06.121873 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121852 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="29410269-90df-41a5-adcf-7d5c3136a321" containerName="registry" Jul 28 18:00:06.121873 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121864 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy" Jul 28 18:00:06.121873 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121869 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121877 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-metric" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121884 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-metric" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121893 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="alertmanager" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121898 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="alertmanager" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121905 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="573e1cdc-5f0a-4078-b2d2-d25231e84224" containerName="console" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121910 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="573e1cdc-5f0a-4078-b2d2-d25231e84224" containerName="console" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121916 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="config-reloader" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121924 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="config-reloader" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121934 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="init-config-reloader" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121940 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="init-config-reloader" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121944 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-web" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121949 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-web" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121962 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="prom-label-proxy" Jul 28 18:00:06.122015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.121968 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="prom-label-proxy" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122034 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="29410269-90df-41a5-adcf-7d5c3136a321" containerName="registry" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122045 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="prom-label-proxy" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122052 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="alertmanager" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122057 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122064 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-metric" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122071 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="config-reloader" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122081 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" containerName="kube-rbac-proxy-web" Jul 28 18:00:06.122389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.122088 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="573e1cdc-5f0a-4078-b2d2-d25231e84224" containerName="console" Jul 28 18:00:06.125792 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.125766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.128724 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.128674 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-v9ggh\"" Jul 28 18:00:06.128986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.128965 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 28 18:00:06.129091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.128970 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 28 18:00:06.129091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.128975 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 28 18:00:06.129236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.129219 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 28 18:00:06.129288 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.129243 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 28 18:00:06.129486 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.129472 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 28 18:00:06.129716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.129679 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 28 18:00:06.130206 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.130193 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 28 18:00:06.135762 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.135742 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 28 18:00:06.146739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.145824 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 18:00:06.190333 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190287 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190333 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190341 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190361 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59009595-f644-4b3b-8305-45a058f2799a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190450 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190496 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190520 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59009595-f644-4b3b-8305-45a058f2799a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190570 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-web-config\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190623 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-config-volume\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190650 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59009595-f644-4b3b-8305-45a058f2799a-config-out\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190679 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190760 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdpsl\" (UniqueName: \"kubernetes.io/projected/59009595-f644-4b3b-8305-45a058f2799a-kube-api-access-pdpsl\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.190842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190818 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59009595-f644-4b3b-8305-45a058f2799a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.191017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.190845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59009595-f644-4b3b-8305-45a058f2799a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291667 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291731 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59009595-f644-4b3b-8305-45a058f2799a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291777 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59009595-f644-4b3b-8305-45a058f2799a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291821 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-web-config\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291856 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-config-volume\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59009595-f644-4b3b-8305-45a058f2799a-config-out\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291911 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.291939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291938 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdpsl\" (UniqueName: \"kubernetes.io/projected/59009595-f644-4b3b-8305-45a058f2799a-kube-api-access-pdpsl\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.292365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.291978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59009595-f644-4b3b-8305-45a058f2799a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.292365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.292001 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59009595-f644-4b3b-8305-45a058f2799a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.292365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.292026 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.292970 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.292942 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/59009595-f644-4b3b-8305-45a058f2799a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.293140 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.293064 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/59009595-f644-4b3b-8305-45a058f2799a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.294944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.294917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/59009595-f644-4b3b-8305-45a058f2799a-config-out\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.294939 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.295000 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.295039 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295223 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.295093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-web-config\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295283 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.295211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/59009595-f644-4b3b-8305-45a058f2799a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295457 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.295431 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.295794 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.295770 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-config-volume\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.296352 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.296334 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/59009595-f644-4b3b-8305-45a058f2799a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.296832 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.296814 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/59009595-f644-4b3b-8305-45a058f2799a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.302448 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.302423 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdpsl\" (UniqueName: \"kubernetes.io/projected/59009595-f644-4b3b-8305-45a058f2799a-kube-api-access-pdpsl\") pod \"alertmanager-main-0\" (UID: \"59009595-f644-4b3b-8305-45a058f2799a\") " pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.437102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.437068 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 28 18:00:06.577314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:06.577224 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 28 18:00:06.579986 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:00:06.579953 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59009595_f644_4b3b_8305_45a058f2799a.slice/crio-6b01bd449ac68f6e95e27534d1516ce23ac5bceddd414a7613c29da8220eb98c WatchSource:0}: Error finding container 6b01bd449ac68f6e95e27534d1516ce23ac5bceddd414a7613c29da8220eb98c: Status 404 returned error can't find the container with id 6b01bd449ac68f6e95e27534d1516ce23ac5bceddd414a7613c29da8220eb98c Jul 28 18:00:07.058410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.058368 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac6deb56-592f-4711-b557-d2eb7ffe2263" path="/var/lib/kubelet/pods/ac6deb56-592f-4711-b557-d2eb7ffe2263/volumes" Jul 28 18:00:07.062583 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.062552 2575 generic.go:356] "Generic (PLEG): container finished" podID="59009595-f644-4b3b-8305-45a058f2799a" containerID="72938ad75bfada1b38028e46bf7f95ea205fa6714abd776f21d8530493a5f37c" exitCode=0 Jul 28 18:00:07.062780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.062589 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerDied","Data":"72938ad75bfada1b38028e46bf7f95ea205fa6714abd776f21d8530493a5f37c"} Jul 28 18:00:07.062780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.062639 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"6b01bd449ac68f6e95e27534d1516ce23ac5bceddd414a7613c29da8220eb98c"} Jul 28 18:00:07.796527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.796489 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6d784b7d86-t954g"] Jul 28 18:00:07.799018 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.798990 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.801613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.801580 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t4hv8\"" Jul 28 18:00:07.802075 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.802050 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 18:00:07.802191 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.802098 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 18:00:07.802402 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.802382 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 18:00:07.802476 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.802401 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 18:00:07.803290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.803270 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 18:00:07.808592 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.808570 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 18:00:07.815604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.815534 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d784b7d86-t954g"] Jul 28 18:00:07.905099 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-serving-certs-ca-bundle\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905099 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-metrics-client-ca\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xps5l\" (UniqueName: \"kubernetes.io/projected/9233ebf0-e875-4fc2-a66c-37d7eed0f203-kube-api-access-xps5l\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905154 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-telemeter-client-tls\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905184 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-federate-client-tls\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905210 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905253 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-secret-telemeter-client\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:07.905349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:07.905284 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006565 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006523 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-metrics-client-ca\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006565 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006569 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xps5l\" (UniqueName: \"kubernetes.io/projected/9233ebf0-e875-4fc2-a66c-37d7eed0f203-kube-api-access-xps5l\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006611 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-telemeter-client-tls\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006638 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-federate-client-tls\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006725 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-secret-telemeter-client\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006750 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.006826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.006815 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-serving-certs-ca-bundle\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.007959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.007928 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-metrics-client-ca\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.008574 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.008311 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-serving-certs-ca-bundle\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.008574 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.008345 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9233ebf0-e875-4fc2-a66c-37d7eed0f203-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.010730 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.010557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.017265 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.016845 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-secret-telemeter-client\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.018069 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.018019 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-federate-client-tls\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.018242 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.018218 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xps5l\" (UniqueName: \"kubernetes.io/projected/9233ebf0-e875-4fc2-a66c-37d7eed0f203-kube-api-access-xps5l\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.019170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.019149 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9233ebf0-e875-4fc2-a66c-37d7eed0f203-telemeter-client-tls\") pod \"telemeter-client-6d784b7d86-t954g\" (UID: \"9233ebf0-e875-4fc2-a66c-37d7eed0f203\") " pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.069654 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.069561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"c6c77e436728fbd07cb8797cc4e8e70cf80a962c447662b50da469f94b60ba90"} Jul 28 18:00:08.070071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.069700 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"93586c2f2416c82939e400e69f5a3442c6dd84b56f03d108af2435b423bd69ea"} Jul 28 18:00:08.070071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.069744 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"2efe5535335c604359ec4eff2b786fb7419085e3d3497f71b8955545c09a6ec6"} Jul 28 18:00:08.070071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.069784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"8e6044703df38979e47c3e3925dc3b2e3b179ebaede4e82d35db6a0e637ad1fe"} Jul 28 18:00:08.070071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.069820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"dc0e9f9d00ecc9895efbf2007dc3f4283c367a8ccd1af39a8d45d1d86355929b"} Jul 28 18:00:08.070071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.069855 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"59009595-f644-4b3b-8305-45a058f2799a","Type":"ContainerStarted","Data":"422c0d2a3fdffb3075887fcf5709f4157415e4049b6993ee3e3d337aac4b5ae8"} Jul 28 18:00:08.109516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.109465 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.093840346 podStartE2EDuration="2.109449514s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.609168ms" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:08.105268912 +0000 UTC m=+205.519925913" watchObservedRunningTime="2026-07-28 18:00:08.109449514 +0000 UTC m=+205.524106516" Jul 28 18:00:08.111762 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.111733 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" Jul 28 18:00:08.267267 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:08.267146 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6d784b7d86-t954g"] Jul 28 18:00:08.271766 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:00:08.271738 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9233ebf0_e875_4fc2_a66c_37d7eed0f203.slice/crio-076432b57291a4b4c289ea7e816de27c33d95c01daa3decbf2bcf206969cc84f WatchSource:0}: Error finding container 076432b57291a4b4c289ea7e816de27c33d95c01daa3decbf2bcf206969cc84f: Status 404 returned error can't find the container with id 076432b57291a4b4c289ea7e816de27c33d95c01daa3decbf2bcf206969cc84f Jul 28 18:00:09.074518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:09.074486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" event={"ID":"9233ebf0-e875-4fc2-a66c-37d7eed0f203","Type":"ContainerStarted","Data":"076432b57291a4b4c289ea7e816de27c33d95c01daa3decbf2bcf206969cc84f"} Jul 28 18:00:11.086505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:11.086471 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" event={"ID":"9233ebf0-e875-4fc2-a66c-37d7eed0f203","Type":"ContainerStarted","Data":"81795ac44027e77754ffc9ba70bdff3d1a9a929e096ce18518e908ea538e965b"} Jul 28 18:00:11.086877 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:11.086540 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" event={"ID":"9233ebf0-e875-4fc2-a66c-37d7eed0f203","Type":"ContainerStarted","Data":"e18bcd0724e3c6a36abc1bd3588875ece2edd20e46e90f26adcc9c0ba2b4b860"} Jul 28 18:00:11.086877 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:11.086561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" event={"ID":"9233ebf0-e875-4fc2-a66c-37d7eed0f203","Type":"ContainerStarted","Data":"755b65bde328c2a56972fdbefa3ae34f90f5267270fe1ee7483a87fe050c2eb1"} Jul 28 18:00:11.111959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:11.111907 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6d784b7d86-t954g" podStartSLOduration=2.306395897 podStartE2EDuration="4.111886641s" totalImagesPullingTime="1.805490744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:11.110614135 +0000 UTC m=+208.525271137" watchObservedRunningTime="2026-07-28 18:00:11.111886641 +0000 UTC m=+208.526543643" Jul 28 18:00:11.989325 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:11.989286 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-79bdf976fb-qsdnx"] Jul 28 18:00:11.991790 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:11.991766 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.004727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.004697 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bdf976fb-qsdnx"] Jul 28 18:00:12.042210 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042176 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-trusted-ca-bundle\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.042210 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042218 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-service-ca\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.042427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042242 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-config\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.042427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042289 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9dl9\" (UniqueName: \"kubernetes.io/projected/3b158da2-1f05-421b-8fc9-347b0fa6ef89-kube-api-access-g9dl9\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.042427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-oauth-serving-cert\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.042427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042410 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-serving-cert\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.042546 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.042446 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-oauth-config\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.143007 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.142957 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-service-ca\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.143497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143134 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-config\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.143497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143175 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9dl9\" (UniqueName: \"kubernetes.io/projected/3b158da2-1f05-421b-8fc9-347b0fa6ef89-kube-api-access-g9dl9\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.143497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143210 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-oauth-serving-cert\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.143806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143784 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-service-ca\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.143894 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143870 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-config\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.144003 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143981 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-serving-cert\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.144058 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.144031 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-oauth-config\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.144118 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.143989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-oauth-serving-cert\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.144118 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.144084 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-trusted-ca-bundle\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.144908 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.144884 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-trusted-ca-bundle\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.146276 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.146246 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-serving-cert\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.146368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.146323 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-oauth-config\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.151854 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.151822 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9dl9\" (UniqueName: \"kubernetes.io/projected/3b158da2-1f05-421b-8fc9-347b0fa6ef89-kube-api-access-g9dl9\") pod \"console-79bdf976fb-qsdnx\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.178877 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.178843 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:12.301771 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.301653 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:12.388328 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.388262 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:12.452843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:12.452810 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-79bdf976fb-qsdnx"] Jul 28 18:00:12.455342 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:00:12.455316 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b158da2_1f05_421b_8fc9_347b0fa6ef89.slice/crio-2d26f6debb9fab25d76d809ae04458b346f9b79751c7a0be4d98e94bbcf59abc WatchSource:0}: Error finding container 2d26f6debb9fab25d76d809ae04458b346f9b79751c7a0be4d98e94bbcf59abc: Status 404 returned error can't find the container with id 2d26f6debb9fab25d76d809ae04458b346f9b79751c7a0be4d98e94bbcf59abc Jul 28 18:00:13.097083 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:13.097049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bdf976fb-qsdnx" event={"ID":"3b158da2-1f05-421b-8fc9-347b0fa6ef89","Type":"ContainerStarted","Data":"c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87"} Jul 28 18:00:13.097284 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:13.097101 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bdf976fb-qsdnx" event={"ID":"3b158da2-1f05-421b-8fc9-347b0fa6ef89","Type":"ContainerStarted","Data":"2d26f6debb9fab25d76d809ae04458b346f9b79751c7a0be4d98e94bbcf59abc"} Jul 28 18:00:13.143526 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:13.143473 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-79bdf976fb-qsdnx" podStartSLOduration=2.143455569 podStartE2EDuration="2.143455569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:00:13.142642043 +0000 UTC m=+210.557299046" watchObservedRunningTime="2026-07-28 18:00:13.143455569 +0000 UTC m=+210.558112570" Jul 28 18:00:13.215796 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:13.215755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:13.282276 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:13.282230 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:22.303000 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:22.302961 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:22.303386 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:22.303056 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:22.307874 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:22.307846 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:23.141379 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:23.141348 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:00:23.233585 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:23.233548 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56689ffbcc-knn76"] Jul 28 18:00:23.405973 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:23.405887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:23.502029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:23.501996 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:23.602948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:23.602917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:43.328485 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:43.328397 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:45.461998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:45.461960 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:45.538524 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:45.538494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:00:48.253171 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.253126 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56689ffbcc-knn76" podUID="f03f4d1a-f727-4bf1-960a-086b85c90445" containerName="console" containerID="cri-o://c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988" gracePeriod=15 Jul 28 18:00:48.487170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.487142 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 18:00:48.658981 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.658944 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-serving-cert\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.658981 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.658981 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-oauth-config\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.659188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659005 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k9qrd\" (UniqueName: \"kubernetes.io/projected/f03f4d1a-f727-4bf1-960a-086b85c90445-kube-api-access-k9qrd\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.659188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659031 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-oauth-serving-cert\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.659188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659122 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-service-ca\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.659188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659157 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-trusted-ca-bundle\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.659386 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-console-config\") pod \"f03f4d1a-f727-4bf1-960a-086b85c90445\" (UID: \"f03f4d1a-f727-4bf1-960a-086b85c90445\") " Jul 28 18:00:48.659677 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659541 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:48.659677 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659651 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-service-ca" (OuterVolumeSpecName: "service-ca") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:48.659900 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659730 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-console-config" (OuterVolumeSpecName: "console-config") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:48.659900 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.659784 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:00:48.661264 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.661237 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03f4d1a-f727-4bf1-960a-086b85c90445-kube-api-access-k9qrd" (OuterVolumeSpecName: "kube-api-access-k9qrd") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "kube-api-access-k9qrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:00:48.661377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.661238 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:48.661377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.661280 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f03f4d1a-f727-4bf1-960a-086b85c90445" (UID: "f03f4d1a-f727-4bf1-960a-086b85c90445"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:00:48.759872 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759823 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-console-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:48.759872 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759866 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:48.759872 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759879 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f03f4d1a-f727-4bf1-960a-086b85c90445-console-oauth-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:48.760117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759895 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9qrd\" (UniqueName: \"kubernetes.io/projected/f03f4d1a-f727-4bf1-960a-086b85c90445-kube-api-access-k9qrd\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:48.760117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759907 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-oauth-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:48.760117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759918 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:48.760117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:48.759931 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f03f4d1a-f727-4bf1-960a-086b85c90445-trusted-ca-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:00:49.248742 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.248678 2575 generic.go:356] "Generic (PLEG): container finished" podID="f03f4d1a-f727-4bf1-960a-086b85c90445" containerID="c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988" exitCode=0 Jul 28 18:00:49.248944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.248748 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56689ffbcc-knn76" Jul 28 18:00:49.248944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.248781 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56689ffbcc-knn76" event={"ID":"f03f4d1a-f727-4bf1-960a-086b85c90445","Type":"ContainerDied","Data":"c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988"} Jul 28 18:00:49.248944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.248838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56689ffbcc-knn76" event={"ID":"f03f4d1a-f727-4bf1-960a-086b85c90445","Type":"ContainerDied","Data":"20334e1420ea6a0894a0e1213b9845f8999bbd273aff2e92cd4de75a10e19455"} Jul 28 18:00:49.248944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.248860 2575 scope.go:117] "RemoveContainer" containerID="c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988" Jul 28 18:00:49.257054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.256881 2575 scope.go:117] "RemoveContainer" containerID="c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988" Jul 28 18:00:49.257297 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:00:49.257201 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988\": container with ID starting with c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988 not found: ID does not exist" containerID="c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988" Jul 28 18:00:49.257297 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.257227 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988"} err="failed to get container status \"c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988\": rpc error: code = NotFound desc = could not find container \"c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988\": container with ID starting with c44f98c9ce863d7dd51575464e07b27bd94e63f56ac084a7946494d472eb8988 not found: ID does not exist" Jul 28 18:00:49.273633 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.273595 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56689ffbcc-knn76"] Jul 28 18:00:49.277526 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:49.277497 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56689ffbcc-knn76"] Jul 28 18:00:51.058350 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:00:51.058316 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03f4d1a-f727-4bf1-960a-086b85c90445" path="/var/lib/kubelet/pods/f03f4d1a-f727-4bf1-960a-086b85c90445/volumes" Jul 28 18:01:17.997547 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:17.997460 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:18.065784 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:18.065752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:21.698087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.698043 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f555dfd6-vjh7g"] Jul 28 18:01:21.698572 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.698538 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03f4d1a-f727-4bf1-960a-086b85c90445" containerName="console" Jul 28 18:01:21.698572 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.698557 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03f4d1a-f727-4bf1-960a-086b85c90445" containerName="console" Jul 28 18:01:21.698727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.698641 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03f4d1a-f727-4bf1-960a-086b85c90445" containerName="console" Jul 28 18:01:21.700779 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.700754 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.721663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.721629 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f555dfd6-vjh7g"] Jul 28 18:01:21.759639 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.759602 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:21.812456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-oauth-serving-cert\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.812456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812472 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-trusted-ca-bundle\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.812738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-serving-cert\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.812738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812635 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-oauth-config\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.812738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-config\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.812867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812741 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-service-ca\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.812867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.812768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59tfg\" (UniqueName: \"kubernetes.io/projected/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-kube-api-access-59tfg\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.828644 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.828603 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:21.913600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913563 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-oauth-config\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.913600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913598 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-config\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.913828 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-service-ca\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.913828 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59tfg\" (UniqueName: \"kubernetes.io/projected/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-kube-api-access-59tfg\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.913828 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913695 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-oauth-serving-cert\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.913962 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913820 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-trusted-ca-bundle\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.913962 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.913882 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-serving-cert\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.914391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.914364 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-config\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.914516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.914495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-service-ca\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.914564 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.914495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-oauth-serving-cert\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.915008 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.914984 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-trusted-ca-bundle\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.916237 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.916215 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-oauth-config\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.916326 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.916309 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-serving-cert\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:21.922010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:21.921979 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59tfg\" (UniqueName: \"kubernetes.io/projected/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-kube-api-access-59tfg\") pod \"console-75f555dfd6-vjh7g\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:22.010339 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:22.010238 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:22.145517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:22.145480 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f555dfd6-vjh7g"] Jul 28 18:01:22.149865 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:01:22.149823 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d8d70f0_1372_4f1c_981a_85f952cb0ecc.slice/crio-f00ae1ea804c86170fc853f6b1b088d3613d9c551dee93d3c37b7a5c96fc58ba WatchSource:0}: Error finding container f00ae1ea804c86170fc853f6b1b088d3613d9c551dee93d3c37b7a5c96fc58ba: Status 404 returned error can't find the container with id f00ae1ea804c86170fc853f6b1b088d3613d9c551dee93d3c37b7a5c96fc58ba Jul 28 18:01:22.381297 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:22.381258 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f555dfd6-vjh7g" event={"ID":"9d8d70f0-1372-4f1c-981a-85f952cb0ecc","Type":"ContainerStarted","Data":"130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd"} Jul 28 18:01:22.381473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:22.381321 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f555dfd6-vjh7g" event={"ID":"9d8d70f0-1372-4f1c-981a-85f952cb0ecc","Type":"ContainerStarted","Data":"f00ae1ea804c86170fc853f6b1b088d3613d9c551dee93d3c37b7a5c96fc58ba"} Jul 28 18:01:22.403358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:22.403305 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75f555dfd6-vjh7g" podStartSLOduration=1.403288463 podStartE2EDuration="1.403288463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:01:22.400286096 +0000 UTC m=+279.814943100" watchObservedRunningTime="2026-07-28 18:01:22.403288463 +0000 UTC m=+279.817945526" Jul 28 18:01:29.984965 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:29.984930 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:30.050216 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:30.050180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:32.010435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.010397 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:32.010917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.010503 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:32.015155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.015128 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:32.433404 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.433375 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:01:32.491575 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.491544 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bdf976fb-qsdnx"] Jul 28 18:01:32.628029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.627993 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:32.728437 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.728345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:32.829741 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:32.829705 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:33.144741 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:33.144706 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:33.241644 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:33.241603 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:42.978735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:42.978703 2575 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 18:01:47.574001 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:47.573963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:01:57.513723 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.513650 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-79bdf976fb-qsdnx" podUID="3b158da2-1f05-421b-8fc9-347b0fa6ef89" containerName="console" containerID="cri-o://c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87" gracePeriod=15 Jul 28 18:01:57.757477 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.757451 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:01:57.938499 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938455 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-config\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.938727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938515 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g9dl9\" (UniqueName: \"kubernetes.io/projected/3b158da2-1f05-421b-8fc9-347b0fa6ef89-kube-api-access-g9dl9\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.938727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938592 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-serving-cert\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.938727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938618 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-oauth-config\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.938727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938643 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-trusted-ca-bundle\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.938727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938663 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-oauth-serving-cert\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.938986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938742 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-service-ca\") pod \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\" (UID: \"3b158da2-1f05-421b-8fc9-347b0fa6ef89\") " Jul 28 18:01:57.939095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.938925 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-config" (OuterVolumeSpecName: "console-config") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:57.939177 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.939152 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:57.939236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.939185 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:57.939290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.939271 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-service-ca" (OuterVolumeSpecName: "service-ca") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:01:57.940851 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.940826 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:01:57.941224 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.941202 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:01:57.941299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:57.941222 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b158da2-1f05-421b-8fc9-347b0fa6ef89-kube-api-access-g9dl9" (OuterVolumeSpecName: "kube-api-access-g9dl9") pod "3b158da2-1f05-421b-8fc9-347b0fa6ef89" (UID: "3b158da2-1f05-421b-8fc9-347b0fa6ef89"). InnerVolumeSpecName "kube-api-access-g9dl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:01:58.040064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040023 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.040064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040056 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9dl9\" (UniqueName: \"kubernetes.io/projected/3b158da2-1f05-421b-8fc9-347b0fa6ef89-kube-api-access-g9dl9\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.040064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040068 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.040293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040080 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3b158da2-1f05-421b-8fc9-347b0fa6ef89-console-oauth-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.040293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040089 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-trusted-ca-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.040293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040097 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-oauth-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.040293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.040106 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3b158da2-1f05-421b-8fc9-347b0fa6ef89-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:01:58.532817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.532777 2575 generic.go:356] "Generic (PLEG): container finished" podID="3b158da2-1f05-421b-8fc9-347b0fa6ef89" containerID="c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87" exitCode=0 Jul 28 18:01:58.532817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.532819 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-79bdf976fb-qsdnx" Jul 28 18:01:58.533250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.532844 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bdf976fb-qsdnx" event={"ID":"3b158da2-1f05-421b-8fc9-347b0fa6ef89","Type":"ContainerDied","Data":"c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87"} Jul 28 18:01:58.533250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.532884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-79bdf976fb-qsdnx" event={"ID":"3b158da2-1f05-421b-8fc9-347b0fa6ef89","Type":"ContainerDied","Data":"2d26f6debb9fab25d76d809ae04458b346f9b79751c7a0be4d98e94bbcf59abc"} Jul 28 18:01:58.533250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.532903 2575 scope.go:117] "RemoveContainer" containerID="c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87" Jul 28 18:01:58.544442 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.544422 2575 scope.go:117] "RemoveContainer" containerID="c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87" Jul 28 18:01:58.544770 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:01:58.544749 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87\": container with ID starting with c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87 not found: ID does not exist" containerID="c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87" Jul 28 18:01:58.544831 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.544778 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87"} err="failed to get container status \"c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87\": rpc error: code = NotFound desc = could not find container \"c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87\": container with ID starting with c1fb7e24887b31a4dad380d84b5cad4061a95f17ffce96267bcd7e271151de87 not found: ID does not exist" Jul 28 18:01:58.556931 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.556904 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-79bdf976fb-qsdnx"] Jul 28 18:01:58.558954 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:58.558930 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-79bdf976fb-qsdnx"] Jul 28 18:01:59.058061 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:01:59.058027 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b158da2-1f05-421b-8fc9-347b0fa6ef89" path="/var/lib/kubelet/pods/3b158da2-1f05-421b-8fc9-347b0fa6ef89/volumes" Jul 28 18:02:26.443348 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:02:26.443308 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:27.659350 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:02:27.659309 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:02:46.710376 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:02:46.710337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:03:45.091736 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.091673 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv"] Jul 28 18:03:45.092218 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.092027 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b158da2-1f05-421b-8fc9-347b0fa6ef89" containerName="console" Jul 28 18:03:45.092218 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.092039 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b158da2-1f05-421b-8fc9-347b0fa6ef89" containerName="console" Jul 28 18:03:45.092218 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.092117 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b158da2-1f05-421b-8fc9-347b0fa6ef89" containerName="console" Jul 28 18:03:45.094161 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.094144 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.097110 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.097056 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 18:03:45.097292 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.097114 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 18:03:45.098083 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.098061 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mqlg8\"" Jul 28 18:03:45.115934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.115893 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv"] Jul 28 18:03:45.171059 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.171012 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwm82\" (UniqueName: \"kubernetes.io/projected/f4a1c02b-21b7-4cdf-aa30-6d75111864be-kube-api-access-qwm82\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.171259 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.171069 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.171259 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.171139 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.272197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.272157 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwm82\" (UniqueName: \"kubernetes.io/projected/f4a1c02b-21b7-4cdf-aa30-6d75111864be-kube-api-access-qwm82\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.272197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.272203 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.272467 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.272226 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.272589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.272558 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.272638 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.272588 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.282274 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.282237 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwm82\" (UniqueName: \"kubernetes.io/projected/f4a1c02b-21b7-4cdf-aa30-6d75111864be-kube-api-access-qwm82\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.403271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.403224 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:03:45.538494 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.538470 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv"] Jul 28 18:03:45.540989 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:03:45.540956 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a1c02b_21b7_4cdf_aa30_6d75111864be.slice/crio-38007f9b39055552b1171b54155e8074a0e97e67e3243cd90014bed6e9db45d9 WatchSource:0}: Error finding container 38007f9b39055552b1171b54155e8074a0e97e67e3243cd90014bed6e9db45d9: Status 404 returned error can't find the container with id 38007f9b39055552b1171b54155e8074a0e97e67e3243cd90014bed6e9db45d9 Jul 28 18:03:45.542891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.542870 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:03:45.955145 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:45.955109 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" event={"ID":"f4a1c02b-21b7-4cdf-aa30-6d75111864be","Type":"ContainerStarted","Data":"38007f9b39055552b1171b54155e8074a0e97e67e3243cd90014bed6e9db45d9"} Jul 28 18:03:54.994564 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:54.994526 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerID="8638f90e1898fac22ee42ac6683a99255e96a162ebb6c619287e04d92312d55a" exitCode=0 Jul 28 18:03:54.995001 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:54.994617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" event={"ID":"f4a1c02b-21b7-4cdf-aa30-6d75111864be","Type":"ContainerDied","Data":"8638f90e1898fac22ee42ac6683a99255e96a162ebb6c619287e04d92312d55a"} Jul 28 18:03:58.010978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:58.010941 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerID="1e359d65acc4f15488bbfc6126d54f36fccacd6f9fc682bea9604a4b8d898552" exitCode=0 Jul 28 18:03:58.011472 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:03:58.011014 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" event={"ID":"f4a1c02b-21b7-4cdf-aa30-6d75111864be","Type":"ContainerDied","Data":"1e359d65acc4f15488bbfc6126d54f36fccacd6f9fc682bea9604a4b8d898552"} Jul 28 18:04:05.047487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:05.047452 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" event={"ID":"f4a1c02b-21b7-4cdf-aa30-6d75111864be","Type":"ContainerStarted","Data":"b4f40cd7a37d355a3bbfe34e4f6531a2f02bf1218b5f5c88b85c560dfc3a299e"} Jul 28 18:04:05.070014 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:05.069966 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" podStartSLOduration=2.139191521 podStartE2EDuration="20.069949141s" totalImagesPullingTime="17.93075762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:03:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:05.067326508 +0000 UTC m=+442.481983510" watchObservedRunningTime="2026-07-28 18:04:05.069949141 +0000 UTC m=+442.484606143" Jul 28 18:04:06.053934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:06.053899 2575 generic.go:356] "Generic (PLEG): container finished" podID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerID="b4f40cd7a37d355a3bbfe34e4f6531a2f02bf1218b5f5c88b85c560dfc3a299e" exitCode=0 Jul 28 18:04:06.054307 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:06.053944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" event={"ID":"f4a1c02b-21b7-4cdf-aa30-6d75111864be","Type":"ContainerDied","Data":"b4f40cd7a37d355a3bbfe34e4f6531a2f02bf1218b5f5c88b85c560dfc3a299e"} Jul 28 18:04:07.236136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.236106 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:04:07.370113 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.370022 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-bundle\") pod \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " Jul 28 18:04:07.370113 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.370062 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-util\") pod \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " Jul 28 18:04:07.370113 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.370101 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwm82\" (UniqueName: \"kubernetes.io/projected/f4a1c02b-21b7-4cdf-aa30-6d75111864be-kube-api-access-qwm82\") pod \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\" (UID: \"f4a1c02b-21b7-4cdf-aa30-6d75111864be\") " Jul 28 18:04:07.370749 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.370722 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-bundle" (OuterVolumeSpecName: "bundle") pod "f4a1c02b-21b7-4cdf-aa30-6d75111864be" (UID: "f4a1c02b-21b7-4cdf-aa30-6d75111864be"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:04:07.372403 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.372379 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a1c02b-21b7-4cdf-aa30-6d75111864be-kube-api-access-qwm82" (OuterVolumeSpecName: "kube-api-access-qwm82") pod "f4a1c02b-21b7-4cdf-aa30-6d75111864be" (UID: "f4a1c02b-21b7-4cdf-aa30-6d75111864be"). InnerVolumeSpecName "kube-api-access-qwm82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:04:07.374410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.374387 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-util" (OuterVolumeSpecName: "util") pod "f4a1c02b-21b7-4cdf-aa30-6d75111864be" (UID: "f4a1c02b-21b7-4cdf-aa30-6d75111864be"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:04:07.471409 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.471371 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:07.471409 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.471402 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f4a1c02b-21b7-4cdf-aa30-6d75111864be-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:07.471409 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:07.471411 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwm82\" (UniqueName: \"kubernetes.io/projected/f4a1c02b-21b7-4cdf-aa30-6d75111864be-kube-api-access-qwm82\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:08.064308 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:08.064276 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" event={"ID":"f4a1c02b-21b7-4cdf-aa30-6d75111864be","Type":"ContainerDied","Data":"38007f9b39055552b1171b54155e8074a0e97e67e3243cd90014bed6e9db45d9"} Jul 28 18:04:08.064308 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:08.064304 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540djxqv" Jul 28 18:04:08.064527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:08.064328 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38007f9b39055552b1171b54155e8074a0e97e67e3243cd90014bed6e9db45d9" Jul 28 18:04:11.206542 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:11.206509 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:12.013664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:12.013626 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:13.411188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411153 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45"] Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411462 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="pull" Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411473 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="pull" Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411484 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="extract" Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411489 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="extract" Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411506 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="util" Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411512 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="util" Jul 28 18:04:13.411560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.411560 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4a1c02b-21b7-4cdf-aa30-6d75111864be" containerName="extract" Jul 28 18:04:13.413632 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.413618 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.416376 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.416353 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 28 18:04:13.416520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.416420 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-nxxmq\"" Jul 28 18:04:13.416578 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.416534 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:04:13.430021 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.429995 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45"] Jul 28 18:04:13.520960 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.520916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec01ae61-b29c-4725-8faf-bf76c88022ac-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lnm45\" (UID: \"ec01ae61-b29c-4725-8faf-bf76c88022ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.520960 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.520961 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m54g\" (UniqueName: \"kubernetes.io/projected/ec01ae61-b29c-4725-8faf-bf76c88022ac-kube-api-access-5m54g\") pod \"cert-manager-operator-controller-manager-55c444bf86-lnm45\" (UID: \"ec01ae61-b29c-4725-8faf-bf76c88022ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.622197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.622162 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec01ae61-b29c-4725-8faf-bf76c88022ac-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lnm45\" (UID: \"ec01ae61-b29c-4725-8faf-bf76c88022ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.622363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.622209 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m54g\" (UniqueName: \"kubernetes.io/projected/ec01ae61-b29c-4725-8faf-bf76c88022ac-kube-api-access-5m54g\") pod \"cert-manager-operator-controller-manager-55c444bf86-lnm45\" (UID: \"ec01ae61-b29c-4725-8faf-bf76c88022ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.622551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.622527 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec01ae61-b29c-4725-8faf-bf76c88022ac-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-lnm45\" (UID: \"ec01ae61-b29c-4725-8faf-bf76c88022ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.632634 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.632611 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m54g\" (UniqueName: \"kubernetes.io/projected/ec01ae61-b29c-4725-8faf-bf76c88022ac-kube-api-access-5m54g\") pod \"cert-manager-operator-controller-manager-55c444bf86-lnm45\" (UID: \"ec01ae61-b29c-4725-8faf-bf76c88022ac\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.722928 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.722833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" Jul 28 18:04:13.861945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:13.861898 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45"] Jul 28 18:04:13.866114 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:13.866083 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec01ae61_b29c_4725_8faf_bf76c88022ac.slice/crio-a17903e2419b27136a30a966e12a666b94425eccab7bd2209d4b775de1bc5c63 WatchSource:0}: Error finding container a17903e2419b27136a30a966e12a666b94425eccab7bd2209d4b775de1bc5c63: Status 404 returned error can't find the container with id a17903e2419b27136a30a966e12a666b94425eccab7bd2209d4b775de1bc5c63 Jul 28 18:04:14.090838 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:14.090736 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" event={"ID":"ec01ae61-b29c-4725-8faf-bf76c88022ac","Type":"ContainerStarted","Data":"a17903e2419b27136a30a966e12a666b94425eccab7bd2209d4b775de1bc5c63"} Jul 28 18:04:14.973860 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:14.973821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:15.070717 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:15.070665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:16.103395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:16.103355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" event={"ID":"ec01ae61-b29c-4725-8faf-bf76c88022ac","Type":"ContainerStarted","Data":"d171a34f69653da582f1b366e1e3a53f15cc8aeaab29737a7ba4526aebcd88f4"} Jul 28 18:04:16.125076 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:16.125023 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-lnm45" podStartSLOduration=1.5781187349999999 podStartE2EDuration="3.125005807s" totalImagesPullingTime="1.546887072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:16.12464194 +0000 UTC m=+453.539298944" watchObservedRunningTime="2026-07-28 18:04:16.125005807 +0000 UTC m=+453.539662809" Jul 28 18:04:16.250950 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:16.250915 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:16.362320 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:16.362231 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:16.486934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:16.486897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:16.620350 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:16.620256 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:17.428132 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.428096 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5"] Jul 28 18:04:17.430636 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.430617 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.433474 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.433443 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 18:04:17.433620 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.433605 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 18:04:17.435812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.435791 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mqlg8\"" Jul 28 18:04:17.444840 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.444811 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5"] Jul 28 18:04:17.557529 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.557495 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.557713 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.557548 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.557713 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.557579 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcbbw\" (UniqueName: \"kubernetes.io/projected/877907f0-da53-418c-8dec-b8454fa7324c-kube-api-access-tcbbw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.658899 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.658860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.658899 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.658907 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcbbw\" (UniqueName: \"kubernetes.io/projected/877907f0-da53-418c-8dec-b8454fa7324c-kube-api-access-tcbbw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.659108 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.658962 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.659331 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.659312 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.659331 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.659315 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.669921 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.669896 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcbbw\" (UniqueName: \"kubernetes.io/projected/877907f0-da53-418c-8dec-b8454fa7324c-kube-api-access-tcbbw\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.739915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.739829 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:17.877610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:17.877585 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5"] Jul 28 18:04:17.879983 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:17.879960 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod877907f0_da53_418c_8dec_b8454fa7324c.slice/crio-10666353442e96b64dd33acd1fe526350db4a4155f1b0c5bb21bbeca621f2da1 WatchSource:0}: Error finding container 10666353442e96b64dd33acd1fe526350db4a4155f1b0c5bb21bbeca621f2da1: Status 404 returned error can't find the container with id 10666353442e96b64dd33acd1fe526350db4a4155f1b0c5bb21bbeca621f2da1 Jul 28 18:04:18.115612 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:18.115485 2575 generic.go:356] "Generic (PLEG): container finished" podID="877907f0-da53-418c-8dec-b8454fa7324c" containerID="a8777262f11fbd4871ac286d152f256a728d8626bb7bf97eed0247411707f903" exitCode=0 Jul 28 18:04:18.115804 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:18.115615 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" event={"ID":"877907f0-da53-418c-8dec-b8454fa7324c","Type":"ContainerDied","Data":"a8777262f11fbd4871ac286d152f256a728d8626bb7bf97eed0247411707f903"} Jul 28 18:04:18.115804 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:18.115712 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" event={"ID":"877907f0-da53-418c-8dec-b8454fa7324c","Type":"ContainerStarted","Data":"10666353442e96b64dd33acd1fe526350db4a4155f1b0c5bb21bbeca621f2da1"} Jul 28 18:04:18.453184 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:18.453152 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:19.055124 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.055087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:19.721107 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.721065 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk"] Jul 28 18:04:19.723515 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.723493 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:19.726045 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.726018 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 28 18:04:19.726193 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.726017 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-qhpgn\"" Jul 28 18:04:19.726991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.726962 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 28 18:04:19.734505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.734472 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk"] Jul 28 18:04:19.877391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.877346 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64ea1be5-33cf-4141-af1a-1e99d3e80bda-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cp6tk\" (UID: \"64ea1be5-33cf-4141-af1a-1e99d3e80bda\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:19.877583 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.877422 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmhc\" (UniqueName: \"kubernetes.io/projected/64ea1be5-33cf-4141-af1a-1e99d3e80bda-kube-api-access-fmmhc\") pod \"cert-manager-cainjector-559cdd8c87-cp6tk\" (UID: \"64ea1be5-33cf-4141-af1a-1e99d3e80bda\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:19.978811 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.978723 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmhc\" (UniqueName: \"kubernetes.io/projected/64ea1be5-33cf-4141-af1a-1e99d3e80bda-kube-api-access-fmmhc\") pod \"cert-manager-cainjector-559cdd8c87-cp6tk\" (UID: \"64ea1be5-33cf-4141-af1a-1e99d3e80bda\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:19.978984 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.978841 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64ea1be5-33cf-4141-af1a-1e99d3e80bda-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cp6tk\" (UID: \"64ea1be5-33cf-4141-af1a-1e99d3e80bda\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:19.988367 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.988330 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/64ea1be5-33cf-4141-af1a-1e99d3e80bda-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-cp6tk\" (UID: \"64ea1be5-33cf-4141-af1a-1e99d3e80bda\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:19.988553 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:19.988379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmhc\" (UniqueName: \"kubernetes.io/projected/64ea1be5-33cf-4141-af1a-1e99d3e80bda-kube-api-access-fmmhc\") pod \"cert-manager-cainjector-559cdd8c87-cp6tk\" (UID: \"64ea1be5-33cf-4141-af1a-1e99d3e80bda\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:20.048025 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:20.047988 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" Jul 28 18:04:20.854476 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:20.854452 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk"] Jul 28 18:04:20.857247 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:20.857222 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64ea1be5_33cf_4141_af1a_1e99d3e80bda.slice/crio-058150bbc55f9bb9ebcb98b3a8c317b059e437f001cfd7b092f2648cc59e47b2 WatchSource:0}: Error finding container 058150bbc55f9bb9ebcb98b3a8c317b059e437f001cfd7b092f2648cc59e47b2: Status 404 returned error can't find the container with id 058150bbc55f9bb9ebcb98b3a8c317b059e437f001cfd7b092f2648cc59e47b2 Jul 28 18:04:21.130955 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:21.130919 2575 generic.go:356] "Generic (PLEG): container finished" podID="877907f0-da53-418c-8dec-b8454fa7324c" containerID="5517fb68f8de9bc52f91968167170a48bf2751f028cbbde46e31230de7e2e381" exitCode=0 Jul 28 18:04:21.131148 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:21.130958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" event={"ID":"877907f0-da53-418c-8dec-b8454fa7324c","Type":"ContainerDied","Data":"5517fb68f8de9bc52f91968167170a48bf2751f028cbbde46e31230de7e2e381"} Jul 28 18:04:21.132317 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:21.132291 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" event={"ID":"64ea1be5-33cf-4141-af1a-1e99d3e80bda","Type":"ContainerStarted","Data":"058150bbc55f9bb9ebcb98b3a8c317b059e437f001cfd7b092f2648cc59e47b2"} Jul 28 18:04:21.391571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:21.391536 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:21.492948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:21.492910 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:22.140550 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:22.140516 2575 generic.go:356] "Generic (PLEG): container finished" podID="877907f0-da53-418c-8dec-b8454fa7324c" containerID="2dd424933f3b9825f54626fbd5e580ae0007994c9c9bbfd10b494907d5e14e12" exitCode=0 Jul 28 18:04:22.141016 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:22.140610 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" event={"ID":"877907f0-da53-418c-8dec-b8454fa7324c","Type":"ContainerDied","Data":"2dd424933f3b9825f54626fbd5e580ae0007994c9c9bbfd10b494907d5e14e12"} Jul 28 18:04:23.962497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:23.962460 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:24.035887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.035854 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xg9mb"] Jul 28 18:04:24.036190 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036178 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="pull" Jul 28 18:04:24.036232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036192 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="pull" Jul 28 18:04:24.036232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036215 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="util" Jul 28 18:04:24.036232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036220 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="util" Jul 28 18:04:24.036232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036229 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="extract" Jul 28 18:04:24.036358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036235 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="extract" Jul 28 18:04:24.036358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.036299 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="877907f0-da53-418c-8dec-b8454fa7324c" containerName="extract" Jul 28 18:04:24.040371 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.040346 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.043176 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.043154 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mvt5s\"" Jul 28 18:04:24.049626 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.049589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xg9mb"] Jul 28 18:04:24.118155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.118125 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tcbbw\" (UniqueName: \"kubernetes.io/projected/877907f0-da53-418c-8dec-b8454fa7324c-kube-api-access-tcbbw\") pod \"877907f0-da53-418c-8dec-b8454fa7324c\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " Jul 28 18:04:24.118323 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.118310 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-bundle\") pod \"877907f0-da53-418c-8dec-b8454fa7324c\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " Jul 28 18:04:24.118368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.118346 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-util\") pod \"877907f0-da53-418c-8dec-b8454fa7324c\" (UID: \"877907f0-da53-418c-8dec-b8454fa7324c\") " Jul 28 18:04:24.118735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.118705 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-bundle" (OuterVolumeSpecName: "bundle") pod "877907f0-da53-418c-8dec-b8454fa7324c" (UID: "877907f0-da53-418c-8dec-b8454fa7324c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:04:24.120279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.120256 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877907f0-da53-418c-8dec-b8454fa7324c-kube-api-access-tcbbw" (OuterVolumeSpecName: "kube-api-access-tcbbw") pod "877907f0-da53-418c-8dec-b8454fa7324c" (UID: "877907f0-da53-418c-8dec-b8454fa7324c"). InnerVolumeSpecName "kube-api-access-tcbbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:04:24.122933 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.122894 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-util" (OuterVolumeSpecName: "util") pod "877907f0-da53-418c-8dec-b8454fa7324c" (UID: "877907f0-da53-418c-8dec-b8454fa7324c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:04:24.153456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.153415 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" event={"ID":"64ea1be5-33cf-4141-af1a-1e99d3e80bda","Type":"ContainerStarted","Data":"a87d9c8f76e3089a3f788e214362eb3de0cd56872efdafdeff4ab833a376586a"} Jul 28 18:04:24.155746 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.155706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" event={"ID":"877907f0-da53-418c-8dec-b8454fa7324c","Type":"ContainerDied","Data":"10666353442e96b64dd33acd1fe526350db4a4155f1b0c5bb21bbeca621f2da1"} Jul 28 18:04:24.155895 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.155763 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10666353442e96b64dd33acd1fe526350db4a4155f1b0c5bb21bbeca621f2da1" Jul 28 18:04:24.155895 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.155823 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fcgdc5" Jul 28 18:04:24.182597 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.182538 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-cp6tk" podStartSLOduration=2.030778529 podStartE2EDuration="5.182518884s" totalImagesPullingTime="3.151740355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:24.180593072 +0000 UTC m=+461.595250074" watchObservedRunningTime="2026-07-28 18:04:24.182518884 +0000 UTC m=+461.597175886" Jul 28 18:04:24.219985 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.219897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3862217a-bd90-46cd-a561-b5d1c4b58401-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xg9mb\" (UID: \"3862217a-bd90-46cd-a561-b5d1c4b58401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.220136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.220023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6ld7\" (UniqueName: \"kubernetes.io/projected/3862217a-bd90-46cd-a561-b5d1c4b58401-kube-api-access-w6ld7\") pod \"cert-manager-webhook-6486ffd997-xg9mb\" (UID: \"3862217a-bd90-46cd-a561-b5d1c4b58401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.220182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.220146 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcbbw\" (UniqueName: \"kubernetes.io/projected/877907f0-da53-418c-8dec-b8454fa7324c-kube-api-access-tcbbw\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:24.220182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.220169 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:24.220245 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.220183 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/877907f0-da53-418c-8dec-b8454fa7324c-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:24.321543 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.321493 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3862217a-bd90-46cd-a561-b5d1c4b58401-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xg9mb\" (UID: \"3862217a-bd90-46cd-a561-b5d1c4b58401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.321781 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.321615 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6ld7\" (UniqueName: \"kubernetes.io/projected/3862217a-bd90-46cd-a561-b5d1c4b58401-kube-api-access-w6ld7\") pod \"cert-manager-webhook-6486ffd997-xg9mb\" (UID: \"3862217a-bd90-46cd-a561-b5d1c4b58401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.330308 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.330271 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/3862217a-bd90-46cd-a561-b5d1c4b58401-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-xg9mb\" (UID: \"3862217a-bd90-46cd-a561-b5d1c4b58401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.330456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.330336 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6ld7\" (UniqueName: \"kubernetes.io/projected/3862217a-bd90-46cd-a561-b5d1c4b58401-kube-api-access-w6ld7\") pod \"cert-manager-webhook-6486ffd997-xg9mb\" (UID: \"3862217a-bd90-46cd-a561-b5d1c4b58401\") " pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.358040 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.358003 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:24.494462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.494377 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-xg9mb"] Jul 28 18:04:24.497460 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:24.497427 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3862217a_bd90_46cd_a561_b5d1c4b58401.slice/crio-61085fa6c47b8ece4ab8ade6b359960546eb2f6d9d9f754d7ef00ef88799e8cf WatchSource:0}: Error finding container 61085fa6c47b8ece4ab8ade6b359960546eb2f6d9d9f754d7ef00ef88799e8cf: Status 404 returned error can't find the container with id 61085fa6c47b8ece4ab8ade6b359960546eb2f6d9d9f754d7ef00ef88799e8cf Jul 28 18:04:24.809813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.809726 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:24.911999 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:24.911958 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:25.162145 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:25.162099 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" event={"ID":"3862217a-bd90-46cd-a561-b5d1c4b58401","Type":"ContainerStarted","Data":"05f6fc0b458cd15308c762e60ff6af5789e9377ba356c89d63eec4cc4e47c86f"} Jul 28 18:04:25.162145 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:25.162149 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" event={"ID":"3862217a-bd90-46cd-a561-b5d1c4b58401","Type":"ContainerStarted","Data":"61085fa6c47b8ece4ab8ade6b359960546eb2f6d9d9f754d7ef00ef88799e8cf"} Jul 28 18:04:25.181510 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:25.181454 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" podStartSLOduration=1.181438289 podStartE2EDuration="1.181438289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:25.180311723 +0000 UTC m=+462.594968726" watchObservedRunningTime="2026-07-28 18:04:25.181438289 +0000 UTC m=+462.596095291" Jul 28 18:04:26.166918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:26.166886 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:27.075784 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:27.075741 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:31.864290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:31.864252 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:31.966357 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:31.966316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:32.173328 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:32.173297 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-xg9mb" Jul 28 18:04:33.846670 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.846631 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk"] Jul 28 18:04:33.897211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.897171 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk"] Jul 28 18:04:33.897383 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.897271 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:33.901863 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.901835 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 28 18:04:33.902037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.901834 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 28 18:04:33.902037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.901950 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-5bw5r\"" Jul 28 18:04:33.994506 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.994464 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28k5k\" (UniqueName: \"kubernetes.io/projected/641a8e7b-0e61-4f49-8ead-4eb6a9c143da-kube-api-access-28k5k\") pod \"openshift-lws-operator-fd8ccff4c-rnslk\" (UID: \"641a8e7b-0e61-4f49-8ead-4eb6a9c143da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:33.994506 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:33.994513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/641a8e7b-0e61-4f49-8ead-4eb6a9c143da-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rnslk\" (UID: \"641a8e7b-0e61-4f49-8ead-4eb6a9c143da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:34.095707 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:34.095640 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28k5k\" (UniqueName: \"kubernetes.io/projected/641a8e7b-0e61-4f49-8ead-4eb6a9c143da-kube-api-access-28k5k\") pod \"openshift-lws-operator-fd8ccff4c-rnslk\" (UID: \"641a8e7b-0e61-4f49-8ead-4eb6a9c143da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:34.095910 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:34.095719 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/641a8e7b-0e61-4f49-8ead-4eb6a9c143da-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rnslk\" (UID: \"641a8e7b-0e61-4f49-8ead-4eb6a9c143da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:34.096138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:34.096111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/641a8e7b-0e61-4f49-8ead-4eb6a9c143da-tmp\") pod \"openshift-lws-operator-fd8ccff4c-rnslk\" (UID: \"641a8e7b-0e61-4f49-8ead-4eb6a9c143da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:34.105513 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:34.105453 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28k5k\" (UniqueName: \"kubernetes.io/projected/641a8e7b-0e61-4f49-8ead-4eb6a9c143da-kube-api-access-28k5k\") pod \"openshift-lws-operator-fd8ccff4c-rnslk\" (UID: \"641a8e7b-0e61-4f49-8ead-4eb6a9c143da\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:34.207676 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:34.207644 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" Jul 28 18:04:34.358812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:34.356851 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk"] Jul 28 18:04:34.360369 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:34.360333 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod641a8e7b_0e61_4f49_8ead_4eb6a9c143da.slice/crio-b641d0eb4d924ce800107aaa710c0ec2afe2db09c9495c2a3d187d667e6d8433 WatchSource:0}: Error finding container b641d0eb4d924ce800107aaa710c0ec2afe2db09c9495c2a3d187d667e6d8433: Status 404 returned error can't find the container with id b641d0eb4d924ce800107aaa710c0ec2afe2db09c9495c2a3d187d667e6d8433 Jul 28 18:04:35.210628 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:35.210593 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" event={"ID":"641a8e7b-0e61-4f49-8ead-4eb6a9c143da","Type":"ContainerStarted","Data":"b641d0eb4d924ce800107aaa710c0ec2afe2db09c9495c2a3d187d667e6d8433"} Jul 28 18:04:37.223402 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:37.223368 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" event={"ID":"641a8e7b-0e61-4f49-8ead-4eb6a9c143da","Type":"ContainerStarted","Data":"fcf6dad07b79e8de2e2e1d217ea7f37ec33272851a6adc7a2d665d1de4a46e91"} Jul 28 18:04:37.243188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:37.243134 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-rnslk" podStartSLOduration=1.881691684 podStartE2EDuration="4.24311856s" totalImagesPullingTime="2.361426876s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:37.241083349 +0000 UTC m=+474.655740351" watchObservedRunningTime="2026-07-28 18:04:37.24311856 +0000 UTC m=+474.657775562" Jul 28 18:04:38.093281 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:38.093249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:38.191865 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:38.191828 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:39.968212 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:39.968172 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:40.322228 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.322135 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj"] Jul 28 18:04:40.324899 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.324870 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.327813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.327784 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 28 18:04:40.328846 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.328825 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 28 18:04:40.328846 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.328842 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-mqlg8\"" Jul 28 18:04:40.337994 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.337963 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj"] Jul 28 18:04:40.350264 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.350229 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-752sp\" (UniqueName: \"kubernetes.io/projected/b4901676-a808-454e-ae17-bb050e70f7e8-kube-api-access-752sp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.350441 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.350282 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.350441 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.350354 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.451795 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.451747 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-752sp\" (UniqueName: \"kubernetes.io/projected/b4901676-a808-454e-ae17-bb050e70f7e8-kube-api-access-752sp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.452004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.451838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.452004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.451876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.452246 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.452224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.452287 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.452245 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.464162 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.464129 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-752sp\" (UniqueName: \"kubernetes.io/projected/b4901676-a808-454e-ae17-bb050e70f7e8-kube-api-access-752sp\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.635038 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.634995 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:40.774973 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:40.774943 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj"] Jul 28 18:04:40.776846 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:40.776819 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4901676_a808_454e_ae17_bb050e70f7e8.slice/crio-bbc22f928825290b87cf0dab15ffb727bdb3e0a6aaf3d398911cbe46105d8e80 WatchSource:0}: Error finding container bbc22f928825290b87cf0dab15ffb727bdb3e0a6aaf3d398911cbe46105d8e80: Status 404 returned error can't find the container with id bbc22f928825290b87cf0dab15ffb727bdb3e0a6aaf3d398911cbe46105d8e80 Jul 28 18:04:41.244502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:41.244465 2575 generic.go:356] "Generic (PLEG): container finished" podID="b4901676-a808-454e-ae17-bb050e70f7e8" containerID="19a5eaa4385fefab6151eb82852cd14d9be2cc21f02843a743736bd2ae91d374" exitCode=0 Jul 28 18:04:41.244999 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:41.244549 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" event={"ID":"b4901676-a808-454e-ae17-bb050e70f7e8","Type":"ContainerDied","Data":"19a5eaa4385fefab6151eb82852cd14d9be2cc21f02843a743736bd2ae91d374"} Jul 28 18:04:41.244999 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:41.244597 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" event={"ID":"b4901676-a808-454e-ae17-bb050e70f7e8","Type":"ContainerStarted","Data":"bbc22f928825290b87cf0dab15ffb727bdb3e0a6aaf3d398911cbe46105d8e80"} Jul 28 18:04:42.252427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:42.252395 2575 generic.go:356] "Generic (PLEG): container finished" podID="b4901676-a808-454e-ae17-bb050e70f7e8" containerID="df6839b4b149ddcb340be81667585ab41359372d437f98619a10df382e357edd" exitCode=0 Jul 28 18:04:42.252945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:42.252441 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" event={"ID":"b4901676-a808-454e-ae17-bb050e70f7e8","Type":"ContainerDied","Data":"df6839b4b149ddcb340be81667585ab41359372d437f98619a10df382e357edd"} Jul 28 18:04:43.015538 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:43.015499 2575 scope.go:117] "RemoveContainer" containerID="650f4302681aef4c0bb0dcc70e24b44d74506321e48c52d4b9c697c5f0c4d619" Jul 28 18:04:43.260406 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:43.260371 2575 generic.go:356] "Generic (PLEG): container finished" podID="b4901676-a808-454e-ae17-bb050e70f7e8" containerID="da47b0a32192313b4c64b1d213ccf07db1795f91de4224fa5ced942019c6e486" exitCode=0 Jul 28 18:04:43.260869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:43.260455 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" event={"ID":"b4901676-a808-454e-ae17-bb050e70f7e8","Type":"ContainerDied","Data":"da47b0a32192313b4c64b1d213ccf07db1795f91de4224fa5ced942019c6e486"} Jul 28 18:04:44.184604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.184567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:44.257759 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.257721 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5"] Jul 28 18:04:44.260941 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.260909 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.264409 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.264384 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 28 18:04:44.264594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.264445 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-q9sjc\"" Jul 28 18:04:44.264594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.264384 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 28 18:04:44.264594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.264475 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 28 18:04:44.272450 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.272422 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5"] Jul 28 18:04:44.280347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.280311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:44.286300 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.286267 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-manager-config\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.286478 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.286322 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gts6t\" (UniqueName: \"kubernetes.io/projected/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-kube-api-access-gts6t\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.286478 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.286389 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-cert\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.286478 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.286413 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-metrics-cert\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.387825 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.387795 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gts6t\" (UniqueName: \"kubernetes.io/projected/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-kube-api-access-gts6t\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.387932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.387891 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-cert\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.387988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.387931 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-metrics-cert\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.387988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.387978 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-manager-config\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.388703 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.388665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-manager-config\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.390697 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.390656 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-metrics-cert\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.390785 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.390715 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-cert\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.396376 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.396347 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gts6t\" (UniqueName: \"kubernetes.io/projected/c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b-kube-api-access-gts6t\") pod \"lws-controller-manager-7cd6475d44-pf9b5\" (UID: \"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b\") " pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.411405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.411371 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:44.488927 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.488837 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-bundle\") pod \"b4901676-a808-454e-ae17-bb050e70f7e8\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " Jul 28 18:04:44.488927 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.488883 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-752sp\" (UniqueName: \"kubernetes.io/projected/b4901676-a808-454e-ae17-bb050e70f7e8-kube-api-access-752sp\") pod \"b4901676-a808-454e-ae17-bb050e70f7e8\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " Jul 28 18:04:44.488927 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.488908 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-util\") pod \"b4901676-a808-454e-ae17-bb050e70f7e8\" (UID: \"b4901676-a808-454e-ae17-bb050e70f7e8\") " Jul 28 18:04:44.489547 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.489521 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-bundle" (OuterVolumeSpecName: "bundle") pod "b4901676-a808-454e-ae17-bb050e70f7e8" (UID: "b4901676-a808-454e-ae17-bb050e70f7e8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:04:44.491007 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.490980 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4901676-a808-454e-ae17-bb050e70f7e8-kube-api-access-752sp" (OuterVolumeSpecName: "kube-api-access-752sp") pod "b4901676-a808-454e-ae17-bb050e70f7e8" (UID: "b4901676-a808-454e-ae17-bb050e70f7e8"). InnerVolumeSpecName "kube-api-access-752sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:04:44.496305 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.496276 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-util" (OuterVolumeSpecName: "util") pod "b4901676-a808-454e-ae17-bb050e70f7e8" (UID: "b4901676-a808-454e-ae17-bb050e70f7e8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:04:44.571929 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.571885 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:44.590356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.590322 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:44.590356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.590353 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-752sp\" (UniqueName: \"kubernetes.io/projected/b4901676-a808-454e-ae17-bb050e70f7e8-kube-api-access-752sp\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:44.590356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.590363 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b4901676-a808-454e-ae17-bb050e70f7e8-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:04:44.722175 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:44.722117 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5"] Jul 28 18:04:44.724312 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:44.724282 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc51ec2af_2948_4ad8_ab2a_d5e1b6fc009b.slice/crio-5932adf3783337d61144da4a4f171c1159c9b30d038ecbea9cb496e8b2705dde WatchSource:0}: Error finding container 5932adf3783337d61144da4a4f171c1159c9b30d038ecbea9cb496e8b2705dde: Status 404 returned error can't find the container with id 5932adf3783337d61144da4a4f171c1159c9b30d038ecbea9cb496e8b2705dde Jul 28 18:04:45.273112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:45.273066 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" event={"ID":"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b","Type":"ContainerStarted","Data":"5932adf3783337d61144da4a4f171c1159c9b30d038ecbea9cb496e8b2705dde"} Jul 28 18:04:45.275016 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:45.274991 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" event={"ID":"b4901676-a808-454e-ae17-bb050e70f7e8","Type":"ContainerDied","Data":"bbc22f928825290b87cf0dab15ffb727bdb3e0a6aaf3d398911cbe46105d8e80"} Jul 28 18:04:45.275138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:45.275019 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afzmwrj" Jul 28 18:04:45.275138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:45.275037 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbc22f928825290b87cf0dab15ffb727bdb3e0a6aaf3d398911cbe46105d8e80" Jul 28 18:04:47.285916 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:47.285881 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" event={"ID":"c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b","Type":"ContainerStarted","Data":"aba039f20cc4cb920fb923558c0220994282648cf3f16cca05fd60029bc5752b"} Jul 28 18:04:47.286317 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:47.285953 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:47.308095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:47.308039 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" podStartSLOduration=1.740134332 podStartE2EDuration="3.30802224s" totalImagesPullingTime="1.567887908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:47.306339417 +0000 UTC m=+484.720996419" watchObservedRunningTime="2026-07-28 18:04:47.30802224 +0000 UTC m=+484.722679242" Jul 28 18:04:51.838358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:51.838324 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:51.955585 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:51.955546 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:52.466341 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:52.466297 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:52.841579 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:52.841490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:52.943111 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:52.943075 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:53.046813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.046774 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44"] Jul 28 18:04:53.047160 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047147 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="extract" Jul 28 18:04:53.047211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047161 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="extract" Jul 28 18:04:53.047211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047175 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="util" Jul 28 18:04:53.047211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047180 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="util" Jul 28 18:04:53.047211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047188 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="pull" Jul 28 18:04:53.047211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047193 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="pull" Jul 28 18:04:53.047359 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.047248 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4901676-a808-454e-ae17-bb050e70f7e8" containerName="extract" Jul 28 18:04:53.049319 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.049299 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.054469 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.054438 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 28 18:04:53.054918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.054888 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 28 18:04:53.054918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.054905 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 28 18:04:53.056337 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.056302 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-f6wlw\"" Jul 28 18:04:53.056831 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.056805 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 28 18:04:53.139946 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.139908 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44"] Jul 28 18:04:53.168596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.168550 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65179d2-f023-42c2-8f23-94791e9afc77-tmp\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.168596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.168594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbjlx\" (UniqueName: \"kubernetes.io/projected/f65179d2-f023-42c2-8f23-94791e9afc77-kube-api-access-fbjlx\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.168859 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.168627 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f65179d2-f023-42c2-8f23-94791e9afc77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.168859 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.168652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f65179d2-f023-42c2-8f23-94791e9afc77-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.168859 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.168738 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f65179d2-f023-42c2-8f23-94791e9afc77-manifests\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.227925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.227887 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:53.269611 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.269567 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65179d2-f023-42c2-8f23-94791e9afc77-tmp\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.269611 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.269612 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbjlx\" (UniqueName: \"kubernetes.io/projected/f65179d2-f023-42c2-8f23-94791e9afc77-kube-api-access-fbjlx\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.269914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.269754 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f65179d2-f023-42c2-8f23-94791e9afc77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.269914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.269798 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f65179d2-f023-42c2-8f23-94791e9afc77-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.269914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.269859 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f65179d2-f023-42c2-8f23-94791e9afc77-manifests\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.270237 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.270211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f65179d2-f023-42c2-8f23-94791e9afc77-manifests\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.272108 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.272077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f65179d2-f023-42c2-8f23-94791e9afc77-tmp\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.272225 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.272211 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f65179d2-f023-42c2-8f23-94791e9afc77-webhook-cert\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.272391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.272370 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/f65179d2-f023-42c2-8f23-94791e9afc77-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.283747 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.283716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbjlx\" (UniqueName: \"kubernetes.io/projected/f65179d2-f023-42c2-8f23-94791e9afc77-kube-api-access-fbjlx\") pod \"opendatahub-operator-controller-manager-5fb95c4cb9-kmp44\" (UID: \"f65179d2-f023-42c2-8f23-94791e9afc77\") " pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.318420 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.318381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:53.361558 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.361512 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:53.421610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.421547 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:53.506661 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.506628 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:53.528829 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.528801 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44"] Jul 28 18:04:53.531572 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:04:53.531548 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65179d2_f023_42c2_8f23_94791e9afc77.slice/crio-ac8c24817a71a6a1929a9256590f830d1185cc6a50b01d3e1869e07d80d83ae6 WatchSource:0}: Error finding container ac8c24817a71a6a1929a9256590f830d1185cc6a50b01d3e1869e07d80d83ae6: Status 404 returned error can't find the container with id ac8c24817a71a6a1929a9256590f830d1185cc6a50b01d3e1869e07d80d83ae6 Jul 28 18:04:53.603010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.602969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:53.810211 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:53.810109 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:04:54.322570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:54.322527 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" event={"ID":"f65179d2-f023-42c2-8f23-94791e9afc77","Type":"ContainerStarted","Data":"ac8c24817a71a6a1929a9256590f830d1185cc6a50b01d3e1869e07d80d83ae6"} Jul 28 18:04:57.341077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:57.341041 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" event={"ID":"f65179d2-f023-42c2-8f23-94791e9afc77","Type":"ContainerStarted","Data":"bae6e0f2015be575498aece63410c10a671b21576124e3b22c8487db84f1598e"} Jul 28 18:04:58.293673 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:58.293640 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7cd6475d44-pf9b5" Jul 28 18:04:58.348023 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:58.347988 2575 generic.go:356] "Generic (PLEG): container finished" podID="f65179d2-f023-42c2-8f23-94791e9afc77" containerID="bae6e0f2015be575498aece63410c10a671b21576124e3b22c8487db84f1598e" exitCode=0 Jul 28 18:04:58.348472 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:58.348075 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" event={"ID":"f65179d2-f023-42c2-8f23-94791e9afc77","Type":"ContainerDied","Data":"bae6e0f2015be575498aece63410c10a671b21576124e3b22c8487db84f1598e"} Jul 28 18:04:59.354847 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:59.354807 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" event={"ID":"f65179d2-f023-42c2-8f23-94791e9afc77","Type":"ContainerStarted","Data":"1ea4e75cd26082f29d5bc42d953fb7c3270eccf8ff53bc1775846f91848d465d"} Jul 28 18:04:59.355235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:59.354954 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:04:59.379907 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:04:59.379847 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" podStartSLOduration=2.055236912 podStartE2EDuration="6.379831764s" totalImagesPullingTime="3.253335328s" totalInitContainerRuntime="1.071259524s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:04:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:04:59.378817677 +0000 UTC m=+496.793474679" watchObservedRunningTime="2026-07-28 18:04:59.379831764 +0000 UTC m=+496.794488765" Jul 28 18:05:06.832140 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:06.832063 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:06.903784 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:06.903752 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:08.812436 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:08.812399 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:08.927342 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:08.927301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:10.362537 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:10.362507 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5fb95c4cb9-kmp44" Jul 28 18:05:12.838379 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:12.838343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:13.132237 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:13.132208 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:13.771419 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:13.771373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:14.043311 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.043203 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb"] Jul 28 18:05:14.050303 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.050265 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.063642 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.063615 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 28 18:05:14.063842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.063715 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 18:05:14.064465 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.064443 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-bdk2z\"" Jul 28 18:05:14.064570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.064505 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 18:05:14.072776 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.072746 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 28 18:05:14.087241 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.087142 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb"] Jul 28 18:05:14.160845 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.160802 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twk69\" (UniqueName: \"kubernetes.io/projected/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-kube-api-access-twk69\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.161037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.160869 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-tls-certs\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.161037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.160915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-tmp\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.261363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.261322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-tls-certs\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.261363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.261369 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-tmp\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.261624 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.261435 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twk69\" (UniqueName: \"kubernetes.io/projected/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-kube-api-access-twk69\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.263802 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.263773 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-tmp\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.263987 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.263966 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-tls-certs\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.299741 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.299632 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twk69\" (UniqueName: \"kubernetes.io/projected/d71a188f-e9d4-490c-902c-e25ef6dfb5b5-kube-api-access-twk69\") pod \"kube-auth-proxy-7cfd4f6bf9-cxdlb\" (UID: \"d71a188f-e9d4-490c-902c-e25ef6dfb5b5\") " pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.359931 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.359894 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" Jul 28 18:05:14.585235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.585136 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:14.619578 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:05:14.619548 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71a188f_e9d4_490c_902c_e25ef6dfb5b5.slice/crio-11f62a37e8f31fea032d560a7cf4bfae7b4ac6aa9bd83b62a4b666591e7ff722 WatchSource:0}: Error finding container 11f62a37e8f31fea032d560a7cf4bfae7b4ac6aa9bd83b62a4b666591e7ff722: Status 404 returned error can't find the container with id 11f62a37e8f31fea032d560a7cf4bfae7b4ac6aa9bd83b62a4b666591e7ff722 Jul 28 18:05:14.656163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:14.656127 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb"] Jul 28 18:05:15.015933 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:15.015895 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:15.272260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:15.272173 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:15.436763 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:15.436706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" event={"ID":"d71a188f-e9d4-490c-902c-e25ef6dfb5b5","Type":"ContainerStarted","Data":"11f62a37e8f31fea032d560a7cf4bfae7b4ac6aa9bd83b62a4b666591e7ff722"} Jul 28 18:05:15.529055 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:15.528951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:16.323521 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:16.323467 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:16.592174 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:16.592087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:18.364035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:18.363985 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:18.452969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:18.452935 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" event={"ID":"d71a188f-e9d4-490c-902c-e25ef6dfb5b5","Type":"ContainerStarted","Data":"5ab19a25e27415fa11935147138ba7c24f7de9e9835d2a5e37ebc9ac385df64d"} Jul 28 18:05:18.488881 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:18.488828 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7cfd4f6bf9-cxdlb" podStartSLOduration=2.4509594789999998 podStartE2EDuration="5.488808949s" totalImagesPullingTime="3.03784947s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:05:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:05:18.486825441 +0000 UTC m=+515.901482444" watchObservedRunningTime="2026-07-28 18:05:18.488808949 +0000 UTC m=+515.903465951" Jul 28 18:05:18.535352 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:18.535318 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:21.177957 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:21.177923 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:23.556637 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:23.556583 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:23.783519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:23.783482 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:24.558892 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:24.558854 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:24.876005 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:24.875916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:25.180075 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:25.180041 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:27.610441 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:27.610400 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:28.141256 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:28.141222 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:28.391372 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:28.391285 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:28.576798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:28.576762 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:05:42.385970 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:05:42.385873 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:09.121824 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:09.121787 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:09.219634 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:09.219594 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:11.752256 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:11.752225 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:11.921364 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:11.921320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:14.398377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:14.398341 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:14.499167 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:14.499134 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:14.629110 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:14.629064 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:15.349076 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:15.349042 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:15.592669 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:15.592627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:27.620079 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:27.620034 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:29.956827 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:29.956795 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn"] Jul 28 18:06:29.960660 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:29.960635 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:29.963756 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:29.963734 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 18:06:29.964548 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:29.964530 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zprdz\"" Jul 28 18:06:29.965532 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:29.965512 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 18:06:29.978365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:29.978336 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn"] Jul 28 18:06:30.019841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.019767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp4g9\" (UniqueName: \"kubernetes.io/projected/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-kube-api-access-xp4g9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.019841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.019835 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.020073 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.019890 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.120741 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.120703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.120915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.120779 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.120915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.120831 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp4g9\" (UniqueName: \"kubernetes.io/projected/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-kube-api-access-xp4g9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.121134 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.121114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.121186 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.121164 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.130718 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.130674 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp4g9\" (UniqueName: \"kubernetes.io/projected/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-kube-api-access-xp4g9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.269667 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.269575 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:30.418513 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.418487 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn"] Jul 28 18:06:30.421051 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:06:30.421026 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9dc5f58_4d70_4e61_8d3c_55dec04a3e63.slice/crio-5cb4e7a4410040121ae49b5a680981953fd4f47c27e4e51cbd16be4fda41d529 WatchSource:0}: Error finding container 5cb4e7a4410040121ae49b5a680981953fd4f47c27e4e51cbd16be4fda41d529: Status 404 returned error can't find the container with id 5cb4e7a4410040121ae49b5a680981953fd4f47c27e4e51cbd16be4fda41d529 Jul 28 18:06:30.550298 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.550261 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j"] Jul 28 18:06:30.560082 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.560052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.573765 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.573726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j"] Jul 28 18:06:30.625416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.625378 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.625583 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.625452 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.625583 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.625473 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv4v\" (UniqueName: \"kubernetes.io/projected/d183ac02-7138-4276-aeed-612fbd2e7b77-kube-api-access-7gv4v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.725966 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.725918 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.725966 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.725966 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv4v\" (UniqueName: \"kubernetes.io/projected/d183ac02-7138-4276-aeed-612fbd2e7b77-kube-api-access-7gv4v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.726174 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.726009 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.726313 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.726294 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.726358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.726320 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.735861 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.735809 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv4v\" (UniqueName: \"kubernetes.io/projected/d183ac02-7138-4276-aeed-612fbd2e7b77-kube-api-access-7gv4v\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.807917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.807834 2575 generic.go:356] "Generic (PLEG): container finished" podID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerID="d4cedebe00cc5f4a68f109b2e50f5ab40d300a55631d42fbb1b7fd309c5f865f" exitCode=0 Jul 28 18:06:30.808068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.807919 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" event={"ID":"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63","Type":"ContainerDied","Data":"d4cedebe00cc5f4a68f109b2e50f5ab40d300a55631d42fbb1b7fd309c5f865f"} Jul 28 18:06:30.808068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.807969 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" event={"ID":"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63","Type":"ContainerStarted","Data":"5cb4e7a4410040121ae49b5a680981953fd4f47c27e4e51cbd16be4fda41d529"} Jul 28 18:06:30.902937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.902896 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:30.922588 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.922553 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x"] Jul 28 18:06:30.926410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.926382 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:30.939322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:30.939287 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x"] Jul 28 18:06:31.028582 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.028541 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zvx6\" (UniqueName: \"kubernetes.io/projected/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-kube-api-access-6zvx6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.028963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.028626 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.028963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.028648 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.047180 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.047120 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j"] Jul 28 18:06:31.049417 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:06:31.049387 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd183ac02_7138_4276_aeed_612fbd2e7b77.slice/crio-b478de6123c03ceb4d3ea0a4a2e53cdec5e3066c8f5ebb15d70e5c7668a40457 WatchSource:0}: Error finding container b478de6123c03ceb4d3ea0a4a2e53cdec5e3066c8f5ebb15d70e5c7668a40457: Status 404 returned error can't find the container with id b478de6123c03ceb4d3ea0a4a2e53cdec5e3066c8f5ebb15d70e5c7668a40457 Jul 28 18:06:31.129830 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.129793 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.129830 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.129832 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.130111 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.129893 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zvx6\" (UniqueName: \"kubernetes.io/projected/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-kube-api-access-6zvx6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.130302 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.130280 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.130377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.130287 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.142830 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.142799 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zvx6\" (UniqueName: \"kubernetes.io/projected/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-kube-api-access-6zvx6\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.236739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.236676 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:31.341276 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.341236 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx"] Jul 28 18:06:31.346718 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.346661 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.361826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.361790 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx"] Jul 28 18:06:31.381893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.381863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x"] Jul 28 18:06:31.384216 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:06:31.384175 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf403b2e1_5fcd_4a34_a161_5ad0795c1c71.slice/crio-90dc061a3a59b1588e7b5f2eb3c6282a4a98aa52e1aea701817a979a13f4a8bd WatchSource:0}: Error finding container 90dc061a3a59b1588e7b5f2eb3c6282a4a98aa52e1aea701817a979a13f4a8bd: Status 404 returned error can't find the container with id 90dc061a3a59b1588e7b5f2eb3c6282a4a98aa52e1aea701817a979a13f4a8bd Jul 28 18:06:31.434785 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.434746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.434963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.434796 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ntv\" (UniqueName: \"kubernetes.io/projected/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-kube-api-access-78ntv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.434963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.434837 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.535756 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.535720 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.535756 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.535760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ntv\" (UniqueName: \"kubernetes.io/projected/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-kube-api-access-78ntv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.536014 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.535787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.536203 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.536179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.536256 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.536224 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.550738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.550665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ntv\" (UniqueName: \"kubernetes.io/projected/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-kube-api-access-78ntv\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.657864 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.657817 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:31.811499 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.811470 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx"] Jul 28 18:06:31.815356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.815326 2575 generic.go:356] "Generic (PLEG): container finished" podID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerID="eeaef01820c4588d0eea54d7246f8035592799079a2295a781a73b23ff0b5207" exitCode=0 Jul 28 18:06:31.815492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.815392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" event={"ID":"f403b2e1-5fcd-4a34-a161-5ad0795c1c71","Type":"ContainerDied","Data":"eeaef01820c4588d0eea54d7246f8035592799079a2295a781a73b23ff0b5207"} Jul 28 18:06:31.815492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.815453 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" event={"ID":"f403b2e1-5fcd-4a34-a161-5ad0795c1c71","Type":"ContainerStarted","Data":"90dc061a3a59b1588e7b5f2eb3c6282a4a98aa52e1aea701817a979a13f4a8bd"} Jul 28 18:06:31.817086 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.817063 2575 generic.go:356] "Generic (PLEG): container finished" podID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerID="386b6bf30cfadd5c6b5a553088480b35647452ccf9235722b40534f7776435b0" exitCode=0 Jul 28 18:06:31.817200 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.817148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" event={"ID":"d183ac02-7138-4276-aeed-612fbd2e7b77","Type":"ContainerDied","Data":"386b6bf30cfadd5c6b5a553088480b35647452ccf9235722b40534f7776435b0"} Jul 28 18:06:31.817263 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.817208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" event={"ID":"d183ac02-7138-4276-aeed-612fbd2e7b77","Type":"ContainerStarted","Data":"b478de6123c03ceb4d3ea0a4a2e53cdec5e3066c8f5ebb15d70e5c7668a40457"} Jul 28 18:06:31.819229 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.819179 2575 generic.go:356] "Generic (PLEG): container finished" podID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerID="4ec42498df5b8b00f8b735cb5df37328c0da8a18e66cf8c39a53a1e686e3a292" exitCode=0 Jul 28 18:06:31.819229 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:31.819209 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" event={"ID":"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63","Type":"ContainerDied","Data":"4ec42498df5b8b00f8b735cb5df37328c0da8a18e66cf8c39a53a1e686e3a292"} Jul 28 18:06:31.830592 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:06:31.830556 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ac4b6b_4c33_4715_bf86_5ed24604d61b.slice/crio-d828334af5c3a8e3aecb6c41d2f6fea86f4c81bbc687be57217efbe0eec30ceb WatchSource:0}: Error finding container d828334af5c3a8e3aecb6c41d2f6fea86f4c81bbc687be57217efbe0eec30ceb: Status 404 returned error can't find the container with id d828334af5c3a8e3aecb6c41d2f6fea86f4c81bbc687be57217efbe0eec30ceb Jul 28 18:06:32.827052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.827016 2575 generic.go:356] "Generic (PLEG): container finished" podID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerID="acd889a4c5339f3b514c6ffeef4fc8aa2cffcf65693a8fed718831855015c79d" exitCode=0 Jul 28 18:06:32.827488 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.827115 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" event={"ID":"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63","Type":"ContainerDied","Data":"acd889a4c5339f3b514c6ffeef4fc8aa2cffcf65693a8fed718831855015c79d"} Jul 28 18:06:32.829001 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.828971 2575 generic.go:356] "Generic (PLEG): container finished" podID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerID="a41cda8d9ecc9ae4d813eefd91341571083d97324b7c5ab3e50984e03211dff1" exitCode=0 Jul 28 18:06:32.829122 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.829055 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" event={"ID":"f403b2e1-5fcd-4a34-a161-5ad0795c1c71","Type":"ContainerDied","Data":"a41cda8d9ecc9ae4d813eefd91341571083d97324b7c5ab3e50984e03211dff1"} Jul 28 18:06:32.830624 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.830599 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerID="6761fa070da18568029fbf6247bc6ac424eec31f95a2b181f6b1ecaf3c2ffb96" exitCode=0 Jul 28 18:06:32.830723 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.830645 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" event={"ID":"c7ac4b6b-4c33-4715-bf86-5ed24604d61b","Type":"ContainerDied","Data":"6761fa070da18568029fbf6247bc6ac424eec31f95a2b181f6b1ecaf3c2ffb96"} Jul 28 18:06:32.830723 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.830714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" event={"ID":"c7ac4b6b-4c33-4715-bf86-5ed24604d61b","Type":"ContainerStarted","Data":"d828334af5c3a8e3aecb6c41d2f6fea86f4c81bbc687be57217efbe0eec30ceb"} Jul 28 18:06:32.832942 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.832916 2575 generic.go:356] "Generic (PLEG): container finished" podID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerID="27dcc181988edcf6f89c1de2ea40f57b3b3f6542402549b3a0540abfa9ab0525" exitCode=0 Jul 28 18:06:32.833064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:32.832995 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" event={"ID":"d183ac02-7138-4276-aeed-612fbd2e7b77","Type":"ContainerDied","Data":"27dcc181988edcf6f89c1de2ea40f57b3b3f6542402549b3a0540abfa9ab0525"} Jul 28 18:06:33.841216 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:33.841177 2575 generic.go:356] "Generic (PLEG): container finished" podID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerID="a7b397abe97a8f81b537fa7730153a9bb7bac7c141b3f5a2918645484cae705b" exitCode=0 Jul 28 18:06:33.841654 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:33.841274 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" event={"ID":"f403b2e1-5fcd-4a34-a161-5ad0795c1c71","Type":"ContainerDied","Data":"a7b397abe97a8f81b537fa7730153a9bb7bac7c141b3f5a2918645484cae705b"} Jul 28 18:06:33.842903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:33.842878 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerID="ae9b87173c3c7bb45b512b4ffe651e092d51e9463ba459c643d2458de2e57856" exitCode=0 Jul 28 18:06:33.842976 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:33.842965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" event={"ID":"c7ac4b6b-4c33-4715-bf86-5ed24604d61b","Type":"ContainerDied","Data":"ae9b87173c3c7bb45b512b4ffe651e092d51e9463ba459c643d2458de2e57856"} Jul 28 18:06:33.845261 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:33.845240 2575 generic.go:356] "Generic (PLEG): container finished" podID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerID="0a06ca21ef81ff5ae0a39a0ddfc751457be1b75021f9e92e52fe3badee3e8a6a" exitCode=0 Jul 28 18:06:33.845351 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:33.845324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" event={"ID":"d183ac02-7138-4276-aeed-612fbd2e7b77","Type":"ContainerDied","Data":"0a06ca21ef81ff5ae0a39a0ddfc751457be1b75021f9e92e52fe3badee3e8a6a"} Jul 28 18:06:34.000780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.000749 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:34.061337 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.061303 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-bundle\") pod \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " Jul 28 18:06:34.061484 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.061353 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-util\") pod \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " Jul 28 18:06:34.061484 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.061375 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xp4g9\" (UniqueName: \"kubernetes.io/projected/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-kube-api-access-xp4g9\") pod \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\" (UID: \"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63\") " Jul 28 18:06:34.061853 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.061823 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-bundle" (OuterVolumeSpecName: "bundle") pod "a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" (UID: "a9dc5f58-4d70-4e61-8d3c-55dec04a3e63"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:34.063574 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.063549 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-kube-api-access-xp4g9" (OuterVolumeSpecName: "kube-api-access-xp4g9") pod "a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" (UID: "a9dc5f58-4d70-4e61-8d3c-55dec04a3e63"). InnerVolumeSpecName "kube-api-access-xp4g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:06:34.066451 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.066413 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-util" (OuterVolumeSpecName: "util") pod "a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" (UID: "a9dc5f58-4d70-4e61-8d3c-55dec04a3e63"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:34.162768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.162650 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:34.162768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.162677 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:34.162768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.162712 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp4g9\" (UniqueName: \"kubernetes.io/projected/a9dc5f58-4d70-4e61-8d3c-55dec04a3e63-kube-api-access-xp4g9\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:34.852874 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.852845 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" Jul 28 18:06:34.853344 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.852875 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn" event={"ID":"a9dc5f58-4d70-4e61-8d3c-55dec04a3e63","Type":"ContainerDied","Data":"5cb4e7a4410040121ae49b5a680981953fd4f47c27e4e51cbd16be4fda41d529"} Jul 28 18:06:34.853344 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.852930 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb4e7a4410040121ae49b5a680981953fd4f47c27e4e51cbd16be4fda41d529" Jul 28 18:06:34.855171 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.855141 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerID="2ffaa8ddf232dc3d101d77727affc7c79c193fa75369e71f4f1bd6cf346bcc14" exitCode=0 Jul 28 18:06:34.855274 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:34.855227 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" event={"ID":"c7ac4b6b-4c33-4715-bf86-5ed24604d61b","Type":"ContainerDied","Data":"2ffaa8ddf232dc3d101d77727affc7c79c193fa75369e71f4f1bd6cf346bcc14"} Jul 28 18:06:35.020910 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.020885 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:35.023289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.023265 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:35.071917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.071885 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gv4v\" (UniqueName: \"kubernetes.io/projected/d183ac02-7138-4276-aeed-612fbd2e7b77-kube-api-access-7gv4v\") pod \"d183ac02-7138-4276-aeed-612fbd2e7b77\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " Jul 28 18:06:35.071917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.071927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6zvx6\" (UniqueName: \"kubernetes.io/projected/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-kube-api-access-6zvx6\") pod \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " Jul 28 18:06:35.071917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.071955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-util\") pod \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " Jul 28 18:06:35.072304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.071990 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-bundle\") pod \"d183ac02-7138-4276-aeed-612fbd2e7b77\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " Jul 28 18:06:35.072304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.072012 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-util\") pod \"d183ac02-7138-4276-aeed-612fbd2e7b77\" (UID: \"d183ac02-7138-4276-aeed-612fbd2e7b77\") " Jul 28 18:06:35.072304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.072140 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-bundle\") pod \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\" (UID: \"f403b2e1-5fcd-4a34-a161-5ad0795c1c71\") " Jul 28 18:06:35.072960 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.072918 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-bundle" (OuterVolumeSpecName: "bundle") pod "d183ac02-7138-4276-aeed-612fbd2e7b77" (UID: "d183ac02-7138-4276-aeed-612fbd2e7b77"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:35.075446 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.075416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-bundle" (OuterVolumeSpecName: "bundle") pod "f403b2e1-5fcd-4a34-a161-5ad0795c1c71" (UID: "f403b2e1-5fcd-4a34-a161-5ad0795c1c71"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:35.075602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.075458 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d183ac02-7138-4276-aeed-612fbd2e7b77-kube-api-access-7gv4v" (OuterVolumeSpecName: "kube-api-access-7gv4v") pod "d183ac02-7138-4276-aeed-612fbd2e7b77" (UID: "d183ac02-7138-4276-aeed-612fbd2e7b77"). InnerVolumeSpecName "kube-api-access-7gv4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:06:35.075717 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.075612 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-kube-api-access-6zvx6" (OuterVolumeSpecName: "kube-api-access-6zvx6") pod "f403b2e1-5fcd-4a34-a161-5ad0795c1c71" (UID: "f403b2e1-5fcd-4a34-a161-5ad0795c1c71"). InnerVolumeSpecName "kube-api-access-6zvx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:06:35.079637 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.079604 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-util" (OuterVolumeSpecName: "util") pod "f403b2e1-5fcd-4a34-a161-5ad0795c1c71" (UID: "f403b2e1-5fcd-4a34-a161-5ad0795c1c71"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:35.080230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.080202 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-util" (OuterVolumeSpecName: "util") pod "d183ac02-7138-4276-aeed-612fbd2e7b77" (UID: "d183ac02-7138-4276-aeed-612fbd2e7b77"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:35.173812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.173762 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:35.173812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.173801 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gv4v\" (UniqueName: \"kubernetes.io/projected/d183ac02-7138-4276-aeed-612fbd2e7b77-kube-api-access-7gv4v\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:35.173812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.173819 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6zvx6\" (UniqueName: \"kubernetes.io/projected/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-kube-api-access-6zvx6\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:35.174056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.173833 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f403b2e1-5fcd-4a34-a161-5ad0795c1c71-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:35.174056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.173847 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:35.174056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.173857 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d183ac02-7138-4276-aeed-612fbd2e7b77-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:35.862664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.862634 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" Jul 28 18:06:35.863102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.862672 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x" event={"ID":"f403b2e1-5fcd-4a34-a161-5ad0795c1c71","Type":"ContainerDied","Data":"90dc061a3a59b1588e7b5f2eb3c6282a4a98aa52e1aea701817a979a13f4a8bd"} Jul 28 18:06:35.863102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.862752 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90dc061a3a59b1588e7b5f2eb3c6282a4a98aa52e1aea701817a979a13f4a8bd" Jul 28 18:06:35.864697 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.864665 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" Jul 28 18:06:35.864697 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.864677 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j" event={"ID":"d183ac02-7138-4276-aeed-612fbd2e7b77","Type":"ContainerDied","Data":"b478de6123c03ceb4d3ea0a4a2e53cdec5e3066c8f5ebb15d70e5c7668a40457"} Jul 28 18:06:35.864852 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:35.864734 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b478de6123c03ceb4d3ea0a4a2e53cdec5e3066c8f5ebb15d70e5c7668a40457" Jul 28 18:06:36.007422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.007395 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:36.081887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.081845 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-util\") pod \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " Jul 28 18:06:36.081887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.081887 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-bundle\") pod \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " Jul 28 18:06:36.082140 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.081927 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78ntv\" (UniqueName: \"kubernetes.io/projected/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-kube-api-access-78ntv\") pod \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\" (UID: \"c7ac4b6b-4c33-4715-bf86-5ed24604d61b\") " Jul 28 18:06:36.082536 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.082510 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-bundle" (OuterVolumeSpecName: "bundle") pod "c7ac4b6b-4c33-4715-bf86-5ed24604d61b" (UID: "c7ac4b6b-4c33-4715-bf86-5ed24604d61b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:36.084058 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.084024 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-kube-api-access-78ntv" (OuterVolumeSpecName: "kube-api-access-78ntv") pod "c7ac4b6b-4c33-4715-bf86-5ed24604d61b" (UID: "c7ac4b6b-4c33-4715-bf86-5ed24604d61b"). InnerVolumeSpecName "kube-api-access-78ntv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:06:36.087385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.087357 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-util" (OuterVolumeSpecName: "util") pod "c7ac4b6b-4c33-4715-bf86-5ed24604d61b" (UID: "c7ac4b6b-4c33-4715-bf86-5ed24604d61b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:06:36.183347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.183308 2575 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-util\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:36.183347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.183341 2575 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:36.183347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.183349 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ntv\" (UniqueName: \"kubernetes.io/projected/c7ac4b6b-4c33-4715-bf86-5ed24604d61b-kube-api-access-78ntv\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:06:36.872132 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.872104 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" Jul 28 18:06:36.872512 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.872138 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx" event={"ID":"c7ac4b6b-4c33-4715-bf86-5ed24604d61b","Type":"ContainerDied","Data":"d828334af5c3a8e3aecb6c41d2f6fea86f4c81bbc687be57217efbe0eec30ceb"} Jul 28 18:06:36.872512 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:36.872182 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d828334af5c3a8e3aecb6c41d2f6fea86f4c81bbc687be57217efbe0eec30ceb" Jul 28 18:06:39.718033 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:39.717999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:43.973831 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:43.973789 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:47.578389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:47.578266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:47.780198 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:47.780161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:52.524087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:52.524045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:52.708530 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:52.708494 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:54.979702 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:54.979659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:55.106430 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:55.106381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:55.892292 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:55.892249 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:57.301561 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:57.301529 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:57.404478 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:57.404437 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:59.673981 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:59.673940 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:06:59.772822 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:06:59.772777 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:01.455434 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:01.455400 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:01.650054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:01.650018 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:02.889166 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:02.889133 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:02.976786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:02.976755 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:08.933121 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:08.933025 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:09.007079 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:09.007045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:12.098756 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.098712 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct"] Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099072 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="util" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099085 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="util" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099099 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="pull" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099104 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="pull" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099113 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="extract" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099118 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="extract" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099126 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="extract" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099131 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="extract" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099138 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="util" Jul 28 18:07:12.099142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099143 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="util" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099150 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="util" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099156 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="util" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099161 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="pull" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099165 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="pull" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099171 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="pull" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099176 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="pull" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099186 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="pull" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099190 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="pull" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099195 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099199 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099206 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="util" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099211 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="util" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099217 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099221 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099268 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f403b2e1-5fcd-4a34-a161-5ad0795c1c71" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099275 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7ac4b6b-4c33-4715-bf86-5ed24604d61b" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099282 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d183ac02-7138-4276-aeed-612fbd2e7b77" containerName="extract" Jul 28 18:07:12.099479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.099287 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9dc5f58-4d70-4e61-8d3c-55dec04a3e63" containerName="extract" Jul 28 18:07:12.102374 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.102349 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:12.107386 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.107361 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 28 18:07:12.108414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.108394 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-ks626\"" Jul 28 18:07:12.108520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.108439 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 28 18:07:12.112304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.112272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtbrh\" (UniqueName: \"kubernetes.io/projected/d9bec5f7-9577-4d87-b050-acd9b6b87490-kube-api-access-dtbrh\") pod \"limitador-operator-controller-manager-859d7b78bd-d2qct\" (UID: \"d9bec5f7-9577-4d87-b050-acd9b6b87490\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:12.125896 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.125863 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct"] Jul 28 18:07:12.213320 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.213267 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtbrh\" (UniqueName: \"kubernetes.io/projected/d9bec5f7-9577-4d87-b050-acd9b6b87490-kube-api-access-dtbrh\") pod \"limitador-operator-controller-manager-859d7b78bd-d2qct\" (UID: \"d9bec5f7-9577-4d87-b050-acd9b6b87490\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:12.226378 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.226346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtbrh\" (UniqueName: \"kubernetes.io/projected/d9bec5f7-9577-4d87-b050-acd9b6b87490-kube-api-access-dtbrh\") pod \"limitador-operator-controller-manager-859d7b78bd-d2qct\" (UID: \"d9bec5f7-9577-4d87-b050-acd9b6b87490\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:12.238954 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.238915 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:12.336801 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.336759 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:12.413701 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.413655 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:12.553406 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:12.553290 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct"] Jul 28 18:07:12.555790 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:07:12.555759 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bec5f7_9577_4d87_b050_acd9b6b87490.slice/crio-e09de435cc4e7013c34ad7e77c7d0e3b0125d191e1fb6eb9958a5a1272e00a38 WatchSource:0}: Error finding container e09de435cc4e7013c34ad7e77c7d0e3b0125d191e1fb6eb9958a5a1272e00a38: Status 404 returned error can't find the container with id e09de435cc4e7013c34ad7e77c7d0e3b0125d191e1fb6eb9958a5a1272e00a38 Jul 28 18:07:13.063943 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:13.063904 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" event={"ID":"d9bec5f7-9577-4d87-b050-acd9b6b87490","Type":"ContainerStarted","Data":"e09de435cc4e7013c34ad7e77c7d0e3b0125d191e1fb6eb9958a5a1272e00a38"} Jul 28 18:07:14.280270 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:14.280233 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:15.078425 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:15.078384 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" event={"ID":"d9bec5f7-9577-4d87-b050-acd9b6b87490","Type":"ContainerStarted","Data":"bce1489d50d194798b5998b2abd93ebd7cc182832082f07420f93fff431cb8b5"} Jul 28 18:07:15.078619 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:15.078457 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:15.100656 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:15.100599 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" podStartSLOduration=1.385891451 podStartE2EDuration="3.100584807s" totalImagesPullingTime="1.714693356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:15.099597896 +0000 UTC m=+632.514254898" watchObservedRunningTime="2026-07-28 18:07:15.100584807 +0000 UTC m=+632.515241808" Jul 28 18:07:15.494541 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:15.494504 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:15.592662 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:15.592619 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:18.067150 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:18.067110 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:18.169217 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:18.169187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:21.066168 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:21.066126 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:21.145380 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:21.145343 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:26.085813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:26.085778 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-d2qct" Jul 28 18:07:28.913987 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:28.913944 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:30.466483 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:30.466444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:35.058884 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:35.058834 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:35.280643 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:35.280597 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:35.458204 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:35.458167 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:35.559787 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:35.559754 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:40.001925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:40.001889 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:40.103577 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:40.103540 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:52.503322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:52.503280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:52.627285 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:52.627247 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:54.236178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:54.236145 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:54.359182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:54.359148 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:54.424154 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:54.424115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:55.071751 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.071704 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-8pgqt"] Jul 28 18:07:55.076902 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.076882 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:07:55.080142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.080117 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ngqrn\"" Jul 28 18:07:55.085204 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.085178 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8pgqt"] Jul 28 18:07:55.101522 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.101485 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gql9m\" (UniqueName: \"kubernetes.io/projected/30640a9c-13f7-4553-b2ef-ab53766a3bd9-kube-api-access-gql9m\") pod \"authorino-58cd955675-8pgqt\" (UID: \"30640a9c-13f7-4553-b2ef-ab53766a3bd9\") " pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:07:55.112117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.112083 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:55.189083 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.189045 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:07:55.202428 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.202396 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gql9m\" (UniqueName: \"kubernetes.io/projected/30640a9c-13f7-4553-b2ef-ab53766a3bd9-kube-api-access-gql9m\") pod \"authorino-58cd955675-8pgqt\" (UID: \"30640a9c-13f7-4553-b2ef-ab53766a3bd9\") " pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:07:55.218053 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.218027 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gql9m\" (UniqueName: \"kubernetes.io/projected/30640a9c-13f7-4553-b2ef-ab53766a3bd9-kube-api-access-gql9m\") pod \"authorino-58cd955675-8pgqt\" (UID: \"30640a9c-13f7-4553-b2ef-ab53766a3bd9\") " pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:07:55.386643 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.386601 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:07:55.521336 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:55.521309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-8pgqt"] Jul 28 18:07:55.524251 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:07:55.524224 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30640a9c_13f7_4553_b2ef_ab53766a3bd9.slice/crio-120c4b6b20823b71bb335b2ed9b227fcb57ba8e258b83ea8a4bd771b510977f2 WatchSource:0}: Error finding container 120c4b6b20823b71bb335b2ed9b227fcb57ba8e258b83ea8a4bd771b510977f2: Status 404 returned error can't find the container with id 120c4b6b20823b71bb335b2ed9b227fcb57ba8e258b83ea8a4bd771b510977f2 Jul 28 18:07:56.291527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:56.291485 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8pgqt" event={"ID":"30640a9c-13f7-4553-b2ef-ab53766a3bd9","Type":"ContainerStarted","Data":"120c4b6b20823b71bb335b2ed9b227fcb57ba8e258b83ea8a4bd771b510977f2"} Jul 28 18:07:59.310914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:59.310874 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8pgqt" event={"ID":"30640a9c-13f7-4553-b2ef-ab53766a3bd9","Type":"ContainerStarted","Data":"2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8"} Jul 28 18:07:59.327058 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:07:59.327005 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-8pgqt" podStartSLOduration=1.114845423 podStartE2EDuration="4.326989197s" totalImagesPullingTime="3.212143774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:07:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:07:59.325956647 +0000 UTC m=+676.740613651" watchObservedRunningTime="2026-07-28 18:07:59.326989197 +0000 UTC m=+676.741646200" Jul 28 18:08:00.300182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:00.300148 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8pgqt"] Jul 28 18:08:01.322300 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:01.322259 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-8pgqt" podUID="30640a9c-13f7-4553-b2ef-ab53766a3bd9" containerName="authorino" containerID="cri-o://2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8" gracePeriod=30 Jul 28 18:08:01.568290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:01.568262 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:08:01.660636 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:01.660593 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gql9m\" (UniqueName: \"kubernetes.io/projected/30640a9c-13f7-4553-b2ef-ab53766a3bd9-kube-api-access-gql9m\") pod \"30640a9c-13f7-4553-b2ef-ab53766a3bd9\" (UID: \"30640a9c-13f7-4553-b2ef-ab53766a3bd9\") " Jul 28 18:08:01.662767 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:01.662724 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30640a9c-13f7-4553-b2ef-ab53766a3bd9-kube-api-access-gql9m" (OuterVolumeSpecName: "kube-api-access-gql9m") pod "30640a9c-13f7-4553-b2ef-ab53766a3bd9" (UID: "30640a9c-13f7-4553-b2ef-ab53766a3bd9"). InnerVolumeSpecName "kube-api-access-gql9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:08:01.762141 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:01.762101 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gql9m\" (UniqueName: \"kubernetes.io/projected/30640a9c-13f7-4553-b2ef-ab53766a3bd9-kube-api-access-gql9m\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:08:02.328928 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.328894 2575 generic.go:356] "Generic (PLEG): container finished" podID="30640a9c-13f7-4553-b2ef-ab53766a3bd9" containerID="2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8" exitCode=0 Jul 28 18:08:02.329377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.328936 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-8pgqt" Jul 28 18:08:02.329377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.328970 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8pgqt" event={"ID":"30640a9c-13f7-4553-b2ef-ab53766a3bd9","Type":"ContainerDied","Data":"2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8"} Jul 28 18:08:02.329377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.329003 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-8pgqt" event={"ID":"30640a9c-13f7-4553-b2ef-ab53766a3bd9","Type":"ContainerDied","Data":"120c4b6b20823b71bb335b2ed9b227fcb57ba8e258b83ea8a4bd771b510977f2"} Jul 28 18:08:02.329377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.329020 2575 scope.go:117] "RemoveContainer" containerID="2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8" Jul 28 18:08:02.337902 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.337881 2575 scope.go:117] "RemoveContainer" containerID="2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8" Jul 28 18:08:02.338188 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:08:02.338170 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8\": container with ID starting with 2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8 not found: ID does not exist" containerID="2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8" Jul 28 18:08:02.338238 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.338196 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8"} err="failed to get container status \"2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8\": rpc error: code = NotFound desc = could not find container \"2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8\": container with ID starting with 2afe73b0f3bf0d732bcace7e63dcfe69d2ad4a7f5a78e513481eb2f8fd39f7d8 not found: ID does not exist" Jul 28 18:08:02.351373 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.351333 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-8pgqt"] Jul 28 18:08:02.355101 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:02.355072 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-8pgqt"] Jul 28 18:08:03.060814 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:03.060782 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30640a9c-13f7-4553-b2ef-ab53766a3bd9" path="/var/lib/kubelet/pods/30640a9c-13f7-4553-b2ef-ab53766a3bd9/volumes" Jul 28 18:08:15.534613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:15.534574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:15.654789 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:15.654751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:16.047068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:16.047032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:28.733184 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.733140 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:28.972190 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.972154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:28.980406 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.980374 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69vb9"] Jul 28 18:08:28.980769 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.980756 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30640a9c-13f7-4553-b2ef-ab53766a3bd9" containerName="authorino" Jul 28 18:08:28.980821 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.980770 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="30640a9c-13f7-4553-b2ef-ab53766a3bd9" containerName="authorino" Jul 28 18:08:28.980857 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.980835 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="30640a9c-13f7-4553-b2ef-ab53766a3bd9" containerName="authorino" Jul 28 18:08:28.984013 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.983971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:28.987283 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.987264 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-ngqrn\"" Jul 28 18:08:28.992697 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:28.992657 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69vb9"] Jul 28 18:08:29.113703 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.113660 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms76g\" (UniqueName: \"kubernetes.io/projected/3df5c66f-3ff5-4e89-a047-7b618d84fd20-kube-api-access-ms76g\") pod \"authorino-5d4b67c446-69vb9\" (UID: \"3df5c66f-3ff5-4e89-a047-7b618d84fd20\") " pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:29.214842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.214803 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms76g\" (UniqueName: \"kubernetes.io/projected/3df5c66f-3ff5-4e89-a047-7b618d84fd20-kube-api-access-ms76g\") pod \"authorino-5d4b67c446-69vb9\" (UID: \"3df5c66f-3ff5-4e89-a047-7b618d84fd20\") " pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:29.238014 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.237943 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms76g\" (UniqueName: \"kubernetes.io/projected/3df5c66f-3ff5-4e89-a047-7b618d84fd20-kube-api-access-ms76g\") pod \"authorino-5d4b67c446-69vb9\" (UID: \"3df5c66f-3ff5-4e89-a047-7b618d84fd20\") " pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:29.293742 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.293674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:29.303523 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.303488 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69vb9"] Jul 28 18:08:29.431044 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.431008 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69vb9"] Jul 28 18:08:29.433064 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:08:29.433032 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3df5c66f_3ff5_4e89_a047_7b618d84fd20.slice/crio-5b1bb58ff15efacf485a1e75ad548378e0d4cc4d106e12e1ca28c738a6e44c50 WatchSource:0}: Error finding container 5b1bb58ff15efacf485a1e75ad548378e0d4cc4d106e12e1ca28c738a6e44c50: Status 404 returned error can't find the container with id 5b1bb58ff15efacf485a1e75ad548378e0d4cc4d106e12e1ca28c738a6e44c50 Jul 28 18:08:29.474570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.474534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-69vb9" event={"ID":"3df5c66f-3ff5-4e89-a047-7b618d84fd20","Type":"ContainerStarted","Data":"5b1bb58ff15efacf485a1e75ad548378e0d4cc4d106e12e1ca28c738a6e44c50"} Jul 28 18:08:29.515106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.515014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:29.515779 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.515760 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6bb5f7666-d7dm7"] Jul 28 18:08:29.520391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.520369 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.525934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.525908 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 28 18:08:29.529280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.529255 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bb5f7666-d7dm7"] Jul 28 18:08:29.618478 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.618428 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f99qd\" (UniqueName: \"kubernetes.io/projected/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-kube-api-access-f99qd\") pod \"authorino-6bb5f7666-d7dm7\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.618666 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.618504 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-tls-cert\") pod \"authorino-6bb5f7666-d7dm7\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.719848 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.719802 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-tls-cert\") pod \"authorino-6bb5f7666-d7dm7\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.720052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.719895 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f99qd\" (UniqueName: \"kubernetes.io/projected/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-kube-api-access-f99qd\") pod \"authorino-6bb5f7666-d7dm7\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.722407 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.722377 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-tls-cert\") pod \"authorino-6bb5f7666-d7dm7\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.730162 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.730134 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f99qd\" (UniqueName: \"kubernetes.io/projected/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-kube-api-access-f99qd\") pod \"authorino-6bb5f7666-d7dm7\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.830139 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.830106 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:08:29.979868 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:29.979842 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6bb5f7666-d7dm7"] Jul 28 18:08:29.981510 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:08:29.981489 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podabe5c47f_e2ef_404a_ae9c_f15f42738bc4.slice/crio-ad44f8bdb56c59023f99d90bc913a8e295e2d41cc7023c1ac0ea2d6e13c83be7 WatchSource:0}: Error finding container ad44f8bdb56c59023f99d90bc913a8e295e2d41cc7023c1ac0ea2d6e13c83be7: Status 404 returned error can't find the container with id ad44f8bdb56c59023f99d90bc913a8e295e2d41cc7023c1ac0ea2d6e13c83be7 Jul 28 18:08:30.483800 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.483755 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" event={"ID":"abe5c47f-e2ef-404a-ae9c-f15f42738bc4","Type":"ContainerStarted","Data":"693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28"} Jul 28 18:08:30.484011 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.483814 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" event={"ID":"abe5c47f-e2ef-404a-ae9c-f15f42738bc4","Type":"ContainerStarted","Data":"ad44f8bdb56c59023f99d90bc913a8e295e2d41cc7023c1ac0ea2d6e13c83be7"} Jul 28 18:08:30.485512 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.485486 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-69vb9" event={"ID":"3df5c66f-3ff5-4e89-a047-7b618d84fd20","Type":"ContainerStarted","Data":"1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96"} Jul 28 18:08:30.485719 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.485641 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-69vb9" podUID="3df5c66f-3ff5-4e89-a047-7b618d84fd20" containerName="authorino" containerID="cri-o://1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96" gracePeriod=30 Jul 28 18:08:30.502423 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.502370 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" podStartSLOduration=1.13338294 podStartE2EDuration="1.502353028s" totalImagesPullingTime="368.970088ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:30.500470955 +0000 UTC m=+707.915127958" watchObservedRunningTime="2026-07-28 18:08:30.502353028 +0000 UTC m=+707.917010031" Jul 28 18:08:30.518613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.518553 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-69vb9" podStartSLOduration=2.160781452 podStartE2EDuration="2.518523687s" totalImagesPullingTime="357.742235ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:08:30.51496546 +0000 UTC m=+707.929622462" watchObservedRunningTime="2026-07-28 18:08:30.518523687 +0000 UTC m=+707.933180690" Jul 28 18:08:30.738135 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.738106 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:30.829117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.829076 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ms76g\" (UniqueName: \"kubernetes.io/projected/3df5c66f-3ff5-4e89-a047-7b618d84fd20-kube-api-access-ms76g\") pod \"3df5c66f-3ff5-4e89-a047-7b618d84fd20\" (UID: \"3df5c66f-3ff5-4e89-a047-7b618d84fd20\") " Jul 28 18:08:30.831157 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.831128 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3df5c66f-3ff5-4e89-a047-7b618d84fd20-kube-api-access-ms76g" (OuterVolumeSpecName: "kube-api-access-ms76g") pod "3df5c66f-3ff5-4e89-a047-7b618d84fd20" (UID: "3df5c66f-3ff5-4e89-a047-7b618d84fd20"). InnerVolumeSpecName "kube-api-access-ms76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:08:30.929714 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:30.929639 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ms76g\" (UniqueName: \"kubernetes.io/projected/3df5c66f-3ff5-4e89-a047-7b618d84fd20-kube-api-access-ms76g\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:08:31.493957 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.493862 2575 generic.go:356] "Generic (PLEG): container finished" podID="3df5c66f-3ff5-4e89-a047-7b618d84fd20" containerID="1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96" exitCode=0 Jul 28 18:08:31.493957 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.493922 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-69vb9" Jul 28 18:08:31.494222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.493951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-69vb9" event={"ID":"3df5c66f-3ff5-4e89-a047-7b618d84fd20","Type":"ContainerDied","Data":"1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96"} Jul 28 18:08:31.494222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.493998 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-69vb9" event={"ID":"3df5c66f-3ff5-4e89-a047-7b618d84fd20","Type":"ContainerDied","Data":"5b1bb58ff15efacf485a1e75ad548378e0d4cc4d106e12e1ca28c738a6e44c50"} Jul 28 18:08:31.494222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.494020 2575 scope.go:117] "RemoveContainer" containerID="1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96" Jul 28 18:08:31.504088 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.504063 2575 scope.go:117] "RemoveContainer" containerID="1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96" Jul 28 18:08:31.504446 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:08:31.504425 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96\": container with ID starting with 1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96 not found: ID does not exist" containerID="1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96" Jul 28 18:08:31.504517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.504456 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96"} err="failed to get container status \"1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96\": rpc error: code = NotFound desc = could not find container \"1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96\": container with ID starting with 1771b869dccd9efc59c49664a1faa8959d1e810b4faf28ee3243a0a25175ca96 not found: ID does not exist" Jul 28 18:08:31.516753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.516711 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69vb9"] Jul 28 18:08:31.520201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:31.520167 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-69vb9"] Jul 28 18:08:33.059864 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:33.059826 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3df5c66f-3ff5-4e89-a047-7b618d84fd20" path="/var/lib/kubelet/pods/3df5c66f-3ff5-4e89-a047-7b618d84fd20/volumes" Jul 28 18:08:35.393504 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:35.393418 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:35.520276 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:35.520235 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:35.610728 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:35.610693 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:35.711363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:35.711263 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:38.541422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:38.541384 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:38.640242 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:38.640205 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:08:44.955362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:08:44.955327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:10.315262 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:10.315224 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:10.402794 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:10.402747 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:10.915511 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:10.915469 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:11.413954 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.413899 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:11.600138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.600100 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:11.708450 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.708345 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:11.894786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.894741 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tprs8"] Jul 28 18:09:11.895189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.895171 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3df5c66f-3ff5-4e89-a047-7b618d84fd20" containerName="authorino" Jul 28 18:09:11.895271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.895191 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3df5c66f-3ff5-4e89-a047-7b618d84fd20" containerName="authorino" Jul 28 18:09:11.895334 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.895321 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3df5c66f-3ff5-4e89-a047-7b618d84fd20" containerName="authorino" Jul 28 18:09:11.898480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.898453 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:11.901161 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.901138 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-qdxsb\"" Jul 28 18:09:11.902215 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.902192 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 28 18:09:11.911648 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.911613 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tprs8"] Jul 28 18:09:11.985990 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.985897 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tprs8\" (UID: \"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:11.986169 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:11.986068 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfksj\" (UniqueName: \"kubernetes.io/projected/43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c-kube-api-access-pfksj\") pod \"payload-processing-5cc6cd444b-tprs8\" (UID: \"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:12.087520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.087482 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfksj\" (UniqueName: \"kubernetes.io/projected/43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c-kube-api-access-pfksj\") pod \"payload-processing-5cc6cd444b-tprs8\" (UID: \"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:12.087520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.087527 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tprs8\" (UID: \"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:12.088263 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.088244 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-tprs8\" (UID: \"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:12.114133 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.114107 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfksj\" (UniqueName: \"kubernetes.io/projected/43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c-kube-api-access-pfksj\") pod \"payload-processing-5cc6cd444b-tprs8\" (UID: \"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c\") " pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:12.210236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.210200 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:12.352461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.352432 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-tprs8"] Jul 28 18:09:12.354425 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:09:12.354393 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43bdcee5_23de_4dcf_899f_fa0ce2dc1d1c.slice/crio-fb4f21060dbd7f2a04a7a792d79489a5eebb9b41c132613b42579b0e1bbb5059 WatchSource:0}: Error finding container fb4f21060dbd7f2a04a7a792d79489a5eebb9b41c132613b42579b0e1bbb5059: Status 404 returned error can't find the container with id fb4f21060dbd7f2a04a7a792d79489a5eebb9b41c132613b42579b0e1bbb5059 Jul 28 18:09:12.356430 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.356410 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:09:12.717051 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:12.717012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" event={"ID":"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c","Type":"ContainerStarted","Data":"fb4f21060dbd7f2a04a7a792d79489a5eebb9b41c132613b42579b0e1bbb5059"} Jul 28 18:09:14.731375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:14.731329 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" event={"ID":"43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c","Type":"ContainerStarted","Data":"bd73685425a7a80bede6a1c3aa718c46d7186950487628d185524a9c0d1a031b"} Jul 28 18:09:14.731822 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:14.731398 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:14.754101 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:14.754029 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" podStartSLOduration=2.153602662 podStartE2EDuration="3.754009585s" totalImagesPullingTime="1.600406923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:09:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:09:14.749896487 +0000 UTC m=+752.164553490" watchObservedRunningTime="2026-07-28 18:09:14.754009585 +0000 UTC m=+752.168666590" Jul 28 18:09:22.048765 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:22.048725 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:22.141745 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:22.141710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:25.739658 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:25.739625 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-tprs8" Jul 28 18:09:37.416223 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:37.416181 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:54.684342 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:54.684299 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:09:59.031005 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:09:59.030963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:10:10.515247 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:10:10.515154 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:10:17.520363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:10:17.520318 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:10:30.428184 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:10:30.428142 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:10:35.877728 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:10:35.877691 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:11:27.937640 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:27.937596 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57cf8ff8f9-nl86t"] Jul 28 18:11:27.941129 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:27.941105 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:27.950879 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:27.950848 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57cf8ff8f9-nl86t"] Jul 28 18:11:27.986214 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:27.986173 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9gc7\" (UniqueName: \"kubernetes.io/projected/bf54ccf4-688f-4d82-9fe3-1fd5996367cc-kube-api-access-w9gc7\") pod \"authorino-57cf8ff8f9-nl86t\" (UID: \"bf54ccf4-688f-4d82-9fe3-1fd5996367cc\") " pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:27.986390 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:27.986247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf54ccf4-688f-4d82-9fe3-1fd5996367cc-tls-cert\") pod \"authorino-57cf8ff8f9-nl86t\" (UID: \"bf54ccf4-688f-4d82-9fe3-1fd5996367cc\") " pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:28.087094 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.087055 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9gc7\" (UniqueName: \"kubernetes.io/projected/bf54ccf4-688f-4d82-9fe3-1fd5996367cc-kube-api-access-w9gc7\") pod \"authorino-57cf8ff8f9-nl86t\" (UID: \"bf54ccf4-688f-4d82-9fe3-1fd5996367cc\") " pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:28.087303 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.087106 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf54ccf4-688f-4d82-9fe3-1fd5996367cc-tls-cert\") pod \"authorino-57cf8ff8f9-nl86t\" (UID: \"bf54ccf4-688f-4d82-9fe3-1fd5996367cc\") " pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:28.089784 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.089757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/bf54ccf4-688f-4d82-9fe3-1fd5996367cc-tls-cert\") pod \"authorino-57cf8ff8f9-nl86t\" (UID: \"bf54ccf4-688f-4d82-9fe3-1fd5996367cc\") " pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:28.096519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.096487 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9gc7\" (UniqueName: \"kubernetes.io/projected/bf54ccf4-688f-4d82-9fe3-1fd5996367cc-kube-api-access-w9gc7\") pod \"authorino-57cf8ff8f9-nl86t\" (UID: \"bf54ccf4-688f-4d82-9fe3-1fd5996367cc\") " pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:28.275207 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.275108 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" Jul 28 18:11:28.345984 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.345948 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:11:28.441169 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:11:28.441141 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf54ccf4_688f_4d82_9fe3_1fd5996367cc.slice/crio-62fc60448872c762a32349f2c68f7c32602b1efa9f7eecf421f08125fd7cb2c2 WatchSource:0}: Error finding container 62fc60448872c762a32349f2c68f7c32602b1efa9f7eecf421f08125fd7cb2c2: Status 404 returned error can't find the container with id 62fc60448872c762a32349f2c68f7c32602b1efa9f7eecf421f08125fd7cb2c2 Jul 28 18:11:28.441291 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.441190 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57cf8ff8f9-nl86t"] Jul 28 18:11:28.525324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:28.525234 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:11:29.446307 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.446257 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" event={"ID":"bf54ccf4-688f-4d82-9fe3-1fd5996367cc","Type":"ContainerStarted","Data":"9331abb2ad8fd2b4540d9dffbca35e5d1d03ce6ca73fd67e97de364e9d3a207e"} Jul 28 18:11:29.446710 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.446324 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" event={"ID":"bf54ccf4-688f-4d82-9fe3-1fd5996367cc","Type":"ContainerStarted","Data":"62fc60448872c762a32349f2c68f7c32602b1efa9f7eecf421f08125fd7cb2c2"} Jul 28 18:11:29.491015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.490957 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57cf8ff8f9-nl86t" podStartSLOduration=2.077097298 podStartE2EDuration="2.490937317s" totalImagesPullingTime="413.840019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:11:29.490373039 +0000 UTC m=+886.905030042" watchObservedRunningTime="2026-07-28 18:11:29.490937317 +0000 UTC m=+886.905594319" Jul 28 18:11:29.561300 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.561251 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bb5f7666-d7dm7"] Jul 28 18:11:29.561785 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.561755 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" podUID="abe5c47f-e2ef-404a-ae9c-f15f42738bc4" containerName="authorino" containerID="cri-o://693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28" gracePeriod=30 Jul 28 18:11:29.567292 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.567256 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:11:29.664296 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.664248 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:11:29.819460 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.819427 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:11:29.903738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.903671 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-tls-cert\") pod \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " Jul 28 18:11:29.903738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.903741 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f99qd\" (UniqueName: \"kubernetes.io/projected/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-kube-api-access-f99qd\") pod \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\" (UID: \"abe5c47f-e2ef-404a-ae9c-f15f42738bc4\") " Jul 28 18:11:29.905926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.905884 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-kube-api-access-f99qd" (OuterVolumeSpecName: "kube-api-access-f99qd") pod "abe5c47f-e2ef-404a-ae9c-f15f42738bc4" (UID: "abe5c47f-e2ef-404a-ae9c-f15f42738bc4"). InnerVolumeSpecName "kube-api-access-f99qd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:11:29.916421 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:29.916385 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "abe5c47f-e2ef-404a-ae9c-f15f42738bc4" (UID: "abe5c47f-e2ef-404a-ae9c-f15f42738bc4"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:11:30.004674 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.004568 2575 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-tls-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:11:30.004674 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.004611 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f99qd\" (UniqueName: \"kubernetes.io/projected/abe5c47f-e2ef-404a-ae9c-f15f42738bc4-kube-api-access-f99qd\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:11:30.453571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.453533 2575 generic.go:356] "Generic (PLEG): container finished" podID="abe5c47f-e2ef-404a-ae9c-f15f42738bc4" containerID="693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28" exitCode=0 Jul 28 18:11:30.454017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.453575 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" Jul 28 18:11:30.454017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.453614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" event={"ID":"abe5c47f-e2ef-404a-ae9c-f15f42738bc4","Type":"ContainerDied","Data":"693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28"} Jul 28 18:11:30.454017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.453652 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6bb5f7666-d7dm7" event={"ID":"abe5c47f-e2ef-404a-ae9c-f15f42738bc4","Type":"ContainerDied","Data":"ad44f8bdb56c59023f99d90bc913a8e295e2d41cc7023c1ac0ea2d6e13c83be7"} Jul 28 18:11:30.454017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.453669 2575 scope.go:117] "RemoveContainer" containerID="693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28" Jul 28 18:11:30.465376 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.465352 2575 scope.go:117] "RemoveContainer" containerID="693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28" Jul 28 18:11:30.465752 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:11:30.465727 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28\": container with ID starting with 693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28 not found: ID does not exist" containerID="693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28" Jul 28 18:11:30.465845 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.465761 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28"} err="failed to get container status \"693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28\": rpc error: code = NotFound desc = could not find container \"693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28\": container with ID starting with 693df1822d2c15cd640a549e3bb74c3bc6f48b6cd008b9556fdb1c83eddf6c28 not found: ID does not exist" Jul 28 18:11:30.480555 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.480515 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6bb5f7666-d7dm7"] Jul 28 18:11:30.493235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:30.489330 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6bb5f7666-d7dm7"] Jul 28 18:11:31.060180 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:31.060140 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="abe5c47f-e2ef-404a-ae9c-f15f42738bc4" path="/var/lib/kubelet/pods/abe5c47f-e2ef-404a-ae9c-f15f42738bc4/volumes" Jul 28 18:11:55.093966 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:55.093927 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:11:55.193531 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:11:55.193497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:12:21.616823 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:12:21.616783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:13:34.677022 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:13:34.676980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:13:35.912299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:13:35.912259 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:16.411275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:16.411214 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:16.511338 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:16.511284 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:16.685266 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:16.685170 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:23.168520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:23.168487 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:23.240401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:23.240367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:23.344727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:23.344674 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:23.457650 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:23.457549 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:23.609535 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:23.609500 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:27.829646 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:27.829610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:27.899040 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:27.898999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:29.138387 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:29.138354 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:29.794982 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:29.794951 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:29.871489 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:29.871452 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:14:35.811597 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:14:35.811497 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:15:00.154841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.154799 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j"] Jul 28 18:15:00.155326 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.155180 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="abe5c47f-e2ef-404a-ae9c-f15f42738bc4" containerName="authorino" Jul 28 18:15:00.155326 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.155191 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="abe5c47f-e2ef-404a-ae9c-f15f42738bc4" containerName="authorino" Jul 28 18:15:00.155326 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.155251 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="abe5c47f-e2ef-404a-ae9c-f15f42738bc4" containerName="authorino" Jul 28 18:15:00.158261 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.158243 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" Jul 28 18:15:00.161222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.161199 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:15:00.167728 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.167694 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j"] Jul 28 18:15:00.169462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.169439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" Jul 28 18:15:00.326136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.326103 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j"] Jul 28 18:15:00.328098 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:15:00.328071 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d25e8a6_cbab_471f_b614_3470df76a572.slice/crio-b3981604e7dce965d102e1d031c21fd3c3a4baa8e634266da4d532f14224cfae WatchSource:0}: Error finding container b3981604e7dce965d102e1d031c21fd3c3a4baa8e634266da4d532f14224cfae: Status 404 returned error can't find the container with id b3981604e7dce965d102e1d031c21fd3c3a4baa8e634266da4d532f14224cfae Jul 28 18:15:00.330126 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.330105 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:15:00.619111 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:00.619019 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" event={"ID":"2d25e8a6-cbab-471f-b614-3470df76a572","Type":"ContainerStarted","Data":"b3981604e7dce965d102e1d031c21fd3c3a4baa8e634266da4d532f14224cfae"} Jul 28 18:15:02.631758 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:02.631714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" event={"ID":"2d25e8a6-cbab-471f-b614-3470df76a572","Type":"ContainerStarted","Data":"0c83180ceb33a272c07c03aa44d8dc47f614c22701babcc2758345288992ed70"} Jul 28 18:15:02.655190 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:02.655132 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" podStartSLOduration=0.732037413 podStartE2EDuration="2.655113468s" totalImagesPullingTime="1.923076055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:15:02.652272855 +0000 UTC m=+1100.066929856" watchObservedRunningTime="2026-07-28 18:15:02.655113468 +0000 UTC m=+1100.069770471" Jul 28 18:15:03.638442 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:03.638400 2575 generic.go:356] "Generic (PLEG): container finished" podID="2d25e8a6-cbab-471f-b614-3470df76a572" containerID="0c83180ceb33a272c07c03aa44d8dc47f614c22701babcc2758345288992ed70" exitCode=0 Jul 28 18:15:03.638878 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:03.638487 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" event={"ID":"2d25e8a6-cbab-471f-b614-3470df76a572","Type":"ContainerDied","Data":"0c83180ceb33a272c07c03aa44d8dc47f614c22701babcc2758345288992ed70"} Jul 28 18:15:04.778136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:04.778103 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" Jul 28 18:15:05.650581 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:05.650543 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" Jul 28 18:15:05.650798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:05.650583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754375-pxr4j" event={"ID":"2d25e8a6-cbab-471f-b614-3470df76a572","Type":"ContainerDied","Data":"b3981604e7dce965d102e1d031c21fd3c3a4baa8e634266da4d532f14224cfae"} Jul 28 18:15:05.650798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:05.650630 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3981604e7dce965d102e1d031c21fd3c3a4baa8e634266da4d532f14224cfae" Jul 28 18:15:12.986770 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:12.986731 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-89c85546f-nz7r8"] Jul 28 18:15:12.987145 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:12.987087 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d25e8a6-cbab-471f-b614-3470df76a572" containerName="cleanup" Jul 28 18:15:12.987145 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:12.987097 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d25e8a6-cbab-471f-b614-3470df76a572" containerName="cleanup" Jul 28 18:15:12.987236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:12.987169 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d25e8a6-cbab-471f-b614-3470df76a572" containerName="cleanup" Jul 28 18:15:12.990271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:12.990253 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.008812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.008778 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89c85546f-nz7r8"] Jul 28 18:15:13.052612 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052573 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-config\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.052844 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052620 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-serving-cert\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.052844 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052767 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-service-ca\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.052844 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052817 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-trusted-ca-bundle\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.052965 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052852 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-oauth-serving-cert\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.052965 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052905 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlc4k\" (UniqueName: \"kubernetes.io/projected/4603e9ff-6845-42b9-9f15-d3c794ffea74-kube-api-access-hlc4k\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.053033 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.052977 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-oauth-config\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.082269 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.082226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:15:13.153741 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153703 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-config\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.153932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153751 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-serving-cert\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.153932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-service-ca\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.153932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153819 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-trusted-ca-bundle\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.153932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153838 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-oauth-serving-cert\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.153932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153857 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlc4k\" (UniqueName: \"kubernetes.io/projected/4603e9ff-6845-42b9-9f15-d3c794ffea74-kube-api-access-hlc4k\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.153932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.153899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-oauth-config\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.154583 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.154557 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-service-ca\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.154726 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.154581 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-config\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.154726 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.154716 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-oauth-serving-cert\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.154845 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.154720 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4603e9ff-6845-42b9-9f15-d3c794ffea74-trusted-ca-bundle\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.156595 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.156565 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-serving-cert\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.156962 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.156940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4603e9ff-6845-42b9-9f15-d3c794ffea74-console-oauth-config\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.163481 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.163456 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlc4k\" (UniqueName: \"kubernetes.io/projected/4603e9ff-6845-42b9-9f15-d3c794ffea74-kube-api-access-hlc4k\") pod \"console-89c85546f-nz7r8\" (UID: \"4603e9ff-6845-42b9-9f15-d3c794ffea74\") " pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.166157 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.166125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:15:13.299839 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.299739 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:13.451579 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.451550 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-89c85546f-nz7r8"] Jul 28 18:15:13.453869 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:15:13.453829 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4603e9ff_6845_42b9_9f15_d3c794ffea74.slice/crio-40a4fb60708c64b10f3081fd4f5699393527cb5e2c557925df3a402ba342ceba WatchSource:0}: Error finding container 40a4fb60708c64b10f3081fd4f5699393527cb5e2c557925df3a402ba342ceba: Status 404 returned error can't find the container with id 40a4fb60708c64b10f3081fd4f5699393527cb5e2c557925df3a402ba342ceba Jul 28 18:15:13.694740 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.694678 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89c85546f-nz7r8" event={"ID":"4603e9ff-6845-42b9-9f15-d3c794ffea74","Type":"ContainerStarted","Data":"a26248ee43f25d4839be09f4c999a20529b74274060985d70c34d7e285f3ba6e"} Jul 28 18:15:13.694937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.694758 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-89c85546f-nz7r8" event={"ID":"4603e9ff-6845-42b9-9f15-d3c794ffea74","Type":"ContainerStarted","Data":"40a4fb60708c64b10f3081fd4f5699393527cb5e2c557925df3a402ba342ceba"} Jul 28 18:15:13.722883 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:13.722819 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-89c85546f-nz7r8" podStartSLOduration=1.722800718 podStartE2EDuration="1.722800718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:15:13.721500507 +0000 UTC m=+1111.136157512" watchObservedRunningTime="2026-07-28 18:15:13.722800718 +0000 UTC m=+1111.137457800" Jul 28 18:15:23.300136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:23.300090 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:23.300568 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:23.300354 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:23.305051 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:23.305027 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:23.754665 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:23.754635 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-89c85546f-nz7r8" Jul 28 18:15:23.808602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:23.808554 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f555dfd6-vjh7g"] Jul 28 18:15:23.923318 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:23.923278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:15:24.024255 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:24.024164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:15:24.091635 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:24.091593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:15:48.828201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:48.828142 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75f555dfd6-vjh7g" podUID="9d8d70f0-1372-4f1c-981a-85f952cb0ecc" containerName="console" containerID="cri-o://130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd" gracePeriod=15 Jul 28 18:15:49.074551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.074526 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:15:49.176062 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176017 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-config\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176068 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-trusted-ca-bundle\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-oauth-serving-cert\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176167 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-serving-cert\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176190 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-oauth-config\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176460 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176336 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-service-ca\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176460 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176413 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59tfg\" (UniqueName: \"kubernetes.io/projected/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-kube-api-access-59tfg\") pod \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\" (UID: \"9d8d70f0-1372-4f1c-981a-85f952cb0ecc\") " Jul 28 18:15:49.176589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176548 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-config" (OuterVolumeSpecName: "console-config") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:15:49.176710 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176663 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-service-ca" (OuterVolumeSpecName: "service-ca") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:15:49.176843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176714 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:15:49.176843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176753 2575 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.176843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176768 2575 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.176843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.176782 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:15:49.178482 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.178456 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-kube-api-access-59tfg" (OuterVolumeSpecName: "kube-api-access-59tfg") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "kube-api-access-59tfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:15:49.178895 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.178874 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:15:49.178969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.178949 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9d8d70f0-1372-4f1c-981a-85f952cb0ecc" (UID: "9d8d70f0-1372-4f1c-981a-85f952cb0ecc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:15:49.277817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.277780 2575 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-oauth-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.277817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.277809 2575 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-serving-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.277817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.277818 2575 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-console-oauth-config\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.277817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.277826 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59tfg\" (UniqueName: \"kubernetes.io/projected/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-kube-api-access-59tfg\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.278096 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.277836 2575 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d8d70f0-1372-4f1c-981a-85f952cb0ecc-trusted-ca-bundle\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:15:49.908602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.908560 2575 generic.go:356] "Generic (PLEG): container finished" podID="9d8d70f0-1372-4f1c-981a-85f952cb0ecc" containerID="130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd" exitCode=0 Jul 28 18:15:49.909091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.908610 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f555dfd6-vjh7g" Jul 28 18:15:49.909091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.908648 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f555dfd6-vjh7g" event={"ID":"9d8d70f0-1372-4f1c-981a-85f952cb0ecc","Type":"ContainerDied","Data":"130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd"} Jul 28 18:15:49.909091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.908714 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f555dfd6-vjh7g" event={"ID":"9d8d70f0-1372-4f1c-981a-85f952cb0ecc","Type":"ContainerDied","Data":"f00ae1ea804c86170fc853f6b1b088d3613d9c551dee93d3c37b7a5c96fc58ba"} Jul 28 18:15:49.909091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.908740 2575 scope.go:117] "RemoveContainer" containerID="130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd" Jul 28 18:15:49.918158 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.918139 2575 scope.go:117] "RemoveContainer" containerID="130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd" Jul 28 18:15:49.918433 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:15:49.918413 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd\": container with ID starting with 130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd not found: ID does not exist" containerID="130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd" Jul 28 18:15:49.918480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.918444 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd"} err="failed to get container status \"130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd\": rpc error: code = NotFound desc = could not find container \"130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd\": container with ID starting with 130e05ff2a5fbc419c4305f38dfa3eb3162401bcd8f7753ae2c893c2d8e7cfcd not found: ID does not exist" Jul 28 18:15:49.932663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.932627 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f555dfd6-vjh7g"] Jul 28 18:15:49.934848 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:49.934825 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f555dfd6-vjh7g"] Jul 28 18:15:51.059051 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:15:51.059013 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d8d70f0-1372-4f1c-981a-85f952cb0ecc" path="/var/lib/kubelet/pods/9d8d70f0-1372-4f1c-981a-85f952cb0ecc/volumes" Jul 28 18:18:56.021257 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:18:56.021215 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:18:58.454525 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:18:58.454490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:21:43.486196 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:21:43.486165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:22:27.848759 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:22:27.848721 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:22:39.734462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:22:39.734428 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:22:59.111643 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:22:59.111608 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:23:16.708391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:23:16.708353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:23:17.446968 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:23:17.446932 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:23:58.934343 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:23:58.934304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:24:00.422215 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:24:00.422155 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:26:35.546868 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:26:35.546782 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:26:58.011419 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:26:58.011382 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:26:58.112198 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:26:58.112161 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:26:59.533079 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:26:59.533032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:02.799911 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:02.799878 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:02.899057 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:02.899016 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:03.276065 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:03.276030 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:09.196332 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.196300 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:09.297454 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.297424 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:09.456664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.456584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:09.761954 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.761883 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9"] Jul 28 18:27:09.762291 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.762279 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d8d70f0-1372-4f1c-981a-85f952cb0ecc" containerName="console" Jul 28 18:27:09.762333 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.762292 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d8d70f0-1372-4f1c-981a-85f952cb0ecc" containerName="console" Jul 28 18:27:09.762368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.762352 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d8d70f0-1372-4f1c-981a-85f952cb0ecc" containerName="console" Jul 28 18:27:09.765383 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.765367 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.767951 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.767925 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 18:27:09.768095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.767976 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0ab65c1c-openshift-default-dockercfg-pschz\"" Jul 28 18:27:09.784629 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.784599 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9"] Jul 28 18:27:09.792191 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.792165 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:09.809905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.809879 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-envoy\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.809913 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-certs\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.809936 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8bsw\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-kube-api-access-m8bsw\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.809970 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-credential-socket\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.810025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-podinfo\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.810047 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-token\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810294 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.810129 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-data\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810294 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.810166 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istiod-ca-cert\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.810294 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.810231 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-socket\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911584 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911540 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-envoy\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911584 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911581 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-certs\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911602 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8bsw\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-kube-api-access-m8bsw\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911631 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-credential-socket\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911730 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-podinfo\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911757 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-token\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911789 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-data\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911810 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istiod-ca-cert\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.911862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.911844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-socket\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.912235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.912085 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-credential-socket\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.912235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.912097 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-certs\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.912235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.912177 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-socket\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.912400 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.912326 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-data\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.912820 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.912796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istiod-ca-cert\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.914087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.914065 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-envoy\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.914320 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.914303 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-podinfo\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.920123 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.920093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-token\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:09.920388 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:09.920365 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8bsw\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-kube-api-access-m8bsw\") pod \"e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:10.076633 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:10.076544 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:10.222862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:10.222833 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9"] Jul 28 18:27:10.225497 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:27:10.225465 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67ac8c6b_0d09_4db0_91bf_9fefee6e86fd.slice/crio-02079092656e4f7420aa056aa7add78b1291077b445c80365bd746cf3ab4da81 WatchSource:0}: Error finding container 02079092656e4f7420aa056aa7add78b1291077b445c80365bd746cf3ab4da81: Status 404 returned error can't find the container with id 02079092656e4f7420aa056aa7add78b1291077b445c80365bd746cf3ab4da81 Jul 28 18:27:10.227696 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:10.227669 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:27:10.243995 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:10.243964 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:10.612216 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:10.612180 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" event={"ID":"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd","Type":"ContainerStarted","Data":"02079092656e4f7420aa056aa7add78b1291077b445c80365bd746cf3ab4da81"} Jul 28 18:27:13.017166 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:13.017123 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:13.017457 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:13.017198 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:13.017457 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:13.017257 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:13.630562 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:13.630525 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" event={"ID":"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd","Type":"ContainerStarted","Data":"4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684"} Jul 28 18:27:13.651622 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:13.651571 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podStartSLOduration=1.8625715170000001 podStartE2EDuration="4.651554931s" totalImagesPullingTime="2.788983414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:13.650316264 +0000 UTC m=+1831.064973267" watchObservedRunningTime="2026-07-28 18:27:13.651554931 +0000 UTC m=+1831.066211933" Jul 28 18:27:14.077181 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:14.077085 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:14.082019 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:14.081993 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:14.636328 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:14.636288 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:14.637268 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:14.637240 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:14.996181 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:14.996100 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:15.099580 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.099539 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:15.174060 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.174021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:15.760011 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.759979 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:15.860335 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.860303 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:15.873078 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.873044 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn"] Jul 28 18:27:15.876448 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.876430 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:15.879307 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.879285 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0ab65c1c\"" Jul 28 18:27:15.891080 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.891053 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn"] Jul 28 18:27:15.910033 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.909995 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92"] Jul 28 18:27:15.913459 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.913439 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:15.916110 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.916082 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0ab65c1c-dockercfg-xzvlc\"" Jul 28 18:27:15.922660 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.922626 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92"] Jul 28 18:27:15.957835 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.957785 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:15.971049 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.971013 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97lv5\" (UniqueName: \"kubernetes.io/projected/f6ec3007-d425-43f0-b4a3-edef2f99c937-kube-api-access-97lv5\") pod \"payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:15.971189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.971060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5csh5\" (UniqueName: \"kubernetes.io/projected/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-kube-api-access-5csh5\") pod \"payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:15.971189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.971106 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:15.971189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:15.971128 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ec3007-d425-43f0-b4a3-edef2f99c937-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.072314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.072218 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5csh5\" (UniqueName: \"kubernetes.io/projected/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-kube-api-access-5csh5\") pod \"payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:16.072314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.072258 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:16.072314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.072278 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ec3007-d425-43f0-b4a3-edef2f99c937-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.072549 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.072361 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97lv5\" (UniqueName: \"kubernetes.io/projected/f6ec3007-d425-43f0-b4a3-edef2f99c937-kube-api-access-97lv5\") pod \"payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.073262 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.073239 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:16.073315 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.073252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ec3007-d425-43f0-b4a3-edef2f99c937-plugins-config-volume\") pod \"payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.080714 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.080668 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97lv5\" (UniqueName: \"kubernetes.io/projected/f6ec3007-d425-43f0-b4a3-edef2f99c937-kube-api-access-97lv5\") pod \"payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.080910 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.080889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5csh5\" (UniqueName: \"kubernetes.io/projected/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-kube-api-access-5csh5\") pod \"payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:16.186999 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.186964 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:16.224625 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.224585 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.336591 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.336553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn"] Jul 28 18:27:16.337280 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:27:16.337247 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a5548b_11e8_40a3_ac4b_afdfe1807f70.slice/crio-e4b3fd806062f9ec22f025b9902844a9b87e32faaef2887f912b7c0f9845f869 WatchSource:0}: Error finding container e4b3fd806062f9ec22f025b9902844a9b87e32faaef2887f912b7c0f9845f869: Status 404 returned error can't find the container with id e4b3fd806062f9ec22f025b9902844a9b87e32faaef2887f912b7c0f9845f869 Jul 28 18:27:16.375123 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.375095 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92"] Jul 28 18:27:16.377873 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:27:16.377846 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6ec3007_d425_43f0_b4a3_edef2f99c937.slice/crio-ca9ba376479d849743ab0e3da283563c3cc0e0327a069fbab878272e3874bf55 WatchSource:0}: Error finding container ca9ba376479d849743ab0e3da283563c3cc0e0327a069fbab878272e3874bf55: Status 404 returned error can't find the container with id ca9ba376479d849743ab0e3da283563c3cc0e0327a069fbab878272e3874bf55 Jul 28 18:27:16.650322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.650284 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" event={"ID":"f6ec3007-d425-43f0-b4a3-edef2f99c937","Type":"ContainerStarted","Data":"dcfc47230b5963d95c9f066f3d19093b128783d11d61a7e0af2d30b369c06bd6"} Jul 28 18:27:16.650535 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.650346 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" event={"ID":"f6ec3007-d425-43f0-b4a3-edef2f99c937","Type":"ContainerStarted","Data":"ca9ba376479d849743ab0e3da283563c3cc0e0327a069fbab878272e3874bf55"} Jul 28 18:27:16.650535 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.650399 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:16.651930 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.651903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" event={"ID":"c7a5548b-11e8-40a3-ac4b-afdfe1807f70","Type":"ContainerStarted","Data":"94c76ccdb1b365c9d347776390e3e3ade2cc33922b23c856dca330c682c3a6b3"} Jul 28 18:27:16.652065 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.651952 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" event={"ID":"c7a5548b-11e8-40a3-ac4b-afdfe1807f70","Type":"ContainerStarted","Data":"e4b3fd806062f9ec22f025b9902844a9b87e32faaef2887f912b7c0f9845f869"} Jul 28 18:27:16.669828 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.669765 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" podStartSLOduration=1.669741873 podStartE2EDuration="1.669741873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:16.667059868 +0000 UTC m=+1834.081716870" watchObservedRunningTime="2026-07-28 18:27:16.669741873 +0000 UTC m=+1834.084398876" Jul 28 18:27:16.684327 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:16.684269 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" podStartSLOduration=1.684250368 podStartE2EDuration="1.684250368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:16.682802283 +0000 UTC m=+1834.097459286" watchObservedRunningTime="2026-07-28 18:27:16.684250368 +0000 UTC m=+1834.098907371" Jul 28 18:27:17.657606 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:17.657575 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:27.660141 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.660109 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:27.716749 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.716706 2575 generic.go:356] "Generic (PLEG): container finished" podID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerID="4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684" exitCode=0 Jul 28 18:27:27.716917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.716784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" event={"ID":"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd","Type":"ContainerDied","Data":"4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684"} Jul 28 18:27:27.717213 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.717198 2575 scope.go:117] "RemoveContainer" containerID="4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684" Jul 28 18:27:27.718292 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.718257 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:27.718351 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.718337 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:27.718388 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:27.718371 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:28.666020 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:28.665990 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:28.724383 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:28.724347 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" event={"ID":"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd","Type":"ContainerStarted","Data":"d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037"} Jul 28 18:27:29.077259 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:29.077168 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:29.078365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:29.078340 2575 patch_prober.go:28] interesting pod/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 28 18:27:29.078479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:29.078389 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 28 18:27:29.636656 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:29.636558 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:30.077160 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:30.077063 2575 patch_prober.go:28] interesting pod/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 28 18:27:30.077160 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:30.077121 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 28 18:27:31.077919 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:31.077885 2575 patch_prober.go:28] interesting pod/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 28 18:27:31.078283 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:31.077942 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 28 18:27:32.077608 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:32.077565 2575 patch_prober.go:28] interesting pod/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 28 18:27:32.077824 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:32.077627 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 28 18:27:33.081804 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:33.081768 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:33.082759 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:33.082738 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:40.585791 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.585749 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp"] Jul 28 18:27:40.593787 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.593734 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:40.596455 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.596425 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:27:40.596596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.596468 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:27:40.596596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.596553 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp"] Jul 28 18:27:40.700657 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.700619 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f0c2aa-620a-4643-ae09-59c57e2d0b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp\" (UID: \"d7f0c2aa-620a-4643-ae09-59c57e2d0b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:40.801937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.801903 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f0c2aa-620a-4643-ae09-59c57e2d0b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp\" (UID: \"d7f0c2aa-620a-4643-ae09-59c57e2d0b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:40.802539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.802517 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f0c2aa-620a-4643-ae09-59c57e2d0b16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp\" (UID: \"d7f0c2aa-620a-4643-ae09-59c57e2d0b16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:40.905964 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:40.905923 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:41.045228 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:41.045195 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp"] Jul 28 18:27:41.046963 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:27:41.046933 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7f0c2aa_620a_4643_ae09_59c57e2d0b16.slice/crio-de72a4474936b255ae59587e9e807b9559de81df62cbdde3260126506f0560b3 WatchSource:0}: Error finding container de72a4474936b255ae59587e9e807b9559de81df62cbdde3260126506f0560b3: Status 404 returned error can't find the container with id de72a4474936b255ae59587e9e807b9559de81df62cbdde3260126506f0560b3 Jul 28 18:27:41.813574 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:41.813532 2575 generic.go:356] "Generic (PLEG): container finished" podID="d7f0c2aa-620a-4643-ae09-59c57e2d0b16" containerID="6fb6c7bcabed0cb8496348fbf316f90d282ed5e92f8e7afb9e4d486e5c651a8a" exitCode=0 Jul 28 18:27:41.813959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:41.813576 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" event={"ID":"d7f0c2aa-620a-4643-ae09-59c57e2d0b16","Type":"ContainerDied","Data":"6fb6c7bcabed0cb8496348fbf316f90d282ed5e92f8e7afb9e4d486e5c651a8a"} Jul 28 18:27:41.813959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:41.813632 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" event={"ID":"d7f0c2aa-620a-4643-ae09-59c57e2d0b16","Type":"ContainerStarted","Data":"de72a4474936b255ae59587e9e807b9559de81df62cbdde3260126506f0560b3"} Jul 28 18:27:42.950482 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:42.950447 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:43.122868 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:43.122766 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f0c2aa-620a-4643-ae09-59c57e2d0b16-maas-api-service-ca\") pod \"d7f0c2aa-620a-4643-ae09-59c57e2d0b16\" (UID: \"d7f0c2aa-620a-4643-ae09-59c57e2d0b16\") " Jul 28 18:27:43.123111 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:43.123086 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7f0c2aa-620a-4643-ae09-59c57e2d0b16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d7f0c2aa-620a-4643-ae09-59c57e2d0b16" (UID: "d7f0c2aa-620a-4643-ae09-59c57e2d0b16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:27:43.224348 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:43.224314 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7f0c2aa-620a-4643-ae09-59c57e2d0b16-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:43.826983 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:43.826944 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" event={"ID":"d7f0c2aa-620a-4643-ae09-59c57e2d0b16","Type":"ContainerDied","Data":"de72a4474936b255ae59587e9e807b9559de81df62cbdde3260126506f0560b3"} Jul 28 18:27:43.826983 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:43.826981 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp" Jul 28 18:27:43.827196 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:43.827022 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de72a4474936b255ae59587e9e807b9559de81df62cbdde3260126506f0560b3" Jul 28 18:27:45.691582 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.691539 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92"] Jul 28 18:27:45.692095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.691976 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" podUID="f6ec3007-d425-43f0-b4a3-edef2f99c937" containerName="payload-processing" containerID="cri-o://dcfc47230b5963d95c9f066f3d19093b128783d11d61a7e0af2d30b369c06bd6" gracePeriod=30 Jul 28 18:27:45.692095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.692070 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:45.699351 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.699314 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn"] Jul 28 18:27:45.699571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.699544 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" podUID="c7a5548b-11e8-40a3-ac4b-afdfe1807f70" containerName="payload-pre-processing" containerID="cri-o://94c76ccdb1b365c9d347776390e3e3ade2cc33922b23c856dca330c682c3a6b3" gracePeriod=30 Jul 28 18:27:45.788668 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.788627 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:45.842390 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.842356 2575 generic.go:356] "Generic (PLEG): container finished" podID="f6ec3007-d425-43f0-b4a3-edef2f99c937" containerID="dcfc47230b5963d95c9f066f3d19093b128783d11d61a7e0af2d30b369c06bd6" exitCode=0 Jul 28 18:27:45.842587 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.842422 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" event={"ID":"f6ec3007-d425-43f0-b4a3-edef2f99c937","Type":"ContainerDied","Data":"dcfc47230b5963d95c9f066f3d19093b128783d11d61a7e0af2d30b369c06bd6"} Jul 28 18:27:45.844509 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.844250 2575 generic.go:356] "Generic (PLEG): container finished" podID="c7a5548b-11e8-40a3-ac4b-afdfe1807f70" containerID="94c76ccdb1b365c9d347776390e3e3ade2cc33922b23c856dca330c682c3a6b3" exitCode=0 Jul 28 18:27:45.844509 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.844358 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" event={"ID":"c7a5548b-11e8-40a3-ac4b-afdfe1807f70","Type":"ContainerDied","Data":"94c76ccdb1b365c9d347776390e3e3ade2cc33922b23c856dca330c682c3a6b3"} Jul 28 18:27:45.855064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.854944 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp"] Jul 28 18:27:45.857375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.857337 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0ab65c1c-mwvkp"] Jul 28 18:27:45.862708 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.862598 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:45.969573 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.969545 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:45.992960 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.992914 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:45.996117 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.996058 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9"] Jul 28 18:27:45.996407 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:45.996373 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" containerID="cri-o://d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037" gracePeriod=30 Jul 28 18:27:46.021807 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.021770 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:46.149414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.149358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97lv5\" (UniqueName: \"kubernetes.io/projected/f6ec3007-d425-43f0-b4a3-edef2f99c937-kube-api-access-97lv5\") pod \"f6ec3007-d425-43f0-b4a3-edef2f99c937\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " Jul 28 18:27:46.149414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.149423 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5csh5\" (UniqueName: \"kubernetes.io/projected/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-kube-api-access-5csh5\") pod \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " Jul 28 18:27:46.149678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.149458 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-plugins-config-volume\") pod \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\" (UID: \"c7a5548b-11e8-40a3-ac4b-afdfe1807f70\") " Jul 28 18:27:46.149678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.149485 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ec3007-d425-43f0-b4a3-edef2f99c937-plugins-config-volume\") pod \"f6ec3007-d425-43f0-b4a3-edef2f99c937\" (UID: \"f6ec3007-d425-43f0-b4a3-edef2f99c937\") " Jul 28 18:27:46.149869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.149843 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c7a5548b-11e8-40a3-ac4b-afdfe1807f70" (UID: "c7a5548b-11e8-40a3-ac4b-afdfe1807f70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:27:46.149915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.149900 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6ec3007-d425-43f0-b4a3-edef2f99c937-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6ec3007-d425-43f0-b4a3-edef2f99c937" (UID: "f6ec3007-d425-43f0-b4a3-edef2f99c937"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:27:46.151768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.151737 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6ec3007-d425-43f0-b4a3-edef2f99c937-kube-api-access-97lv5" (OuterVolumeSpecName: "kube-api-access-97lv5") pod "f6ec3007-d425-43f0-b4a3-edef2f99c937" (UID: "f6ec3007-d425-43f0-b4a3-edef2f99c937"). InnerVolumeSpecName "kube-api-access-97lv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:27:46.151856 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.151746 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-kube-api-access-5csh5" (OuterVolumeSpecName: "kube-api-access-5csh5") pod "c7a5548b-11e8-40a3-ac4b-afdfe1807f70" (UID: "c7a5548b-11e8-40a3-ac4b-afdfe1807f70"). InnerVolumeSpecName "kube-api-access-5csh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:27:46.250604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.250492 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97lv5\" (UniqueName: \"kubernetes.io/projected/f6ec3007-d425-43f0-b4a3-edef2f99c937-kube-api-access-97lv5\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:46.250604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.250535 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5csh5\" (UniqueName: \"kubernetes.io/projected/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-kube-api-access-5csh5\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:46.250604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.250548 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7a5548b-11e8-40a3-ac4b-afdfe1807f70-plugins-config-volume\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:46.250604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.250561 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6ec3007-d425-43f0-b4a3-edef2f99c937-plugins-config-volume\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:46.267974 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.267881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:46.555121 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.554964 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:46.655954 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.655903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:46.789021 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.788990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:46.851134 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.851038 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" Jul 28 18:27:46.851134 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.851046 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92" event={"ID":"f6ec3007-d425-43f0-b4a3-edef2f99c937","Type":"ContainerDied","Data":"ca9ba376479d849743ab0e3da283563c3cc0e0327a069fbab878272e3874bf55"} Jul 28 18:27:46.851372 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.851188 2575 scope.go:117] "RemoveContainer" containerID="dcfc47230b5963d95c9f066f3d19093b128783d11d61a7e0af2d30b369c06bd6" Jul 28 18:27:46.852560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.852499 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" Jul 28 18:27:46.852560 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.852534 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn" event={"ID":"c7a5548b-11e8-40a3-ac4b-afdfe1807f70","Type":"ContainerDied","Data":"e4b3fd806062f9ec22f025b9902844a9b87e32faaef2887f912b7c0f9845f869"} Jul 28 18:27:46.862293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.862271 2575 scope.go:117] "RemoveContainer" containerID="94c76ccdb1b365c9d347776390e3e3ade2cc33922b23c856dca330c682c3a6b3" Jul 28 18:27:46.876898 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.876861 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92"] Jul 28 18:27:46.882073 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.882039 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0ab65c1c-6554466fc4-wls92"] Jul 28 18:27:46.895421 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.895385 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn"] Jul 28 18:27:46.898390 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:46.898361 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0ab65c1c-85d48b956-kv6wn"] Jul 28 18:27:47.061369 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.061335 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c7a5548b-11e8-40a3-ac4b-afdfe1807f70" path="/var/lib/kubelet/pods/c7a5548b-11e8-40a3-ac4b-afdfe1807f70/volumes" Jul 28 18:27:47.061677 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.061664 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7f0c2aa-620a-4643-ae09-59c57e2d0b16" path="/var/lib/kubelet/pods/d7f0c2aa-620a-4643-ae09-59c57e2d0b16/volumes" Jul 28 18:27:47.062006 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.061993 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6ec3007-d425-43f0-b4a3-edef2f99c937" path="/var/lib/kubelet/pods/f6ec3007-d425-43f0-b4a3-edef2f99c937/volumes" Jul 28 18:27:47.164924 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.164886 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh"] Jul 28 18:27:47.165317 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165303 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c7a5548b-11e8-40a3-ac4b-afdfe1807f70" containerName="payload-pre-processing" Jul 28 18:27:47.165317 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165318 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="c7a5548b-11e8-40a3-ac4b-afdfe1807f70" containerName="payload-pre-processing" Jul 28 18:27:47.165408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165337 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7f0c2aa-620a-4643-ae09-59c57e2d0b16" containerName="revoke-keys" Jul 28 18:27:47.165408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165342 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7f0c2aa-620a-4643-ae09-59c57e2d0b16" containerName="revoke-keys" Jul 28 18:27:47.165408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165352 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6ec3007-d425-43f0-b4a3-edef2f99c937" containerName="payload-processing" Jul 28 18:27:47.165408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165362 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6ec3007-d425-43f0-b4a3-edef2f99c937" containerName="payload-processing" Jul 28 18:27:47.165523 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165425 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7f0c2aa-620a-4643-ae09-59c57e2d0b16" containerName="revoke-keys" Jul 28 18:27:47.165523 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165440 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6ec3007-d425-43f0-b4a3-edef2f99c937" containerName="payload-processing" Jul 28 18:27:47.165523 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.165451 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="c7a5548b-11e8-40a3-ac4b-afdfe1807f70" containerName="payload-pre-processing" Jul 28 18:27:47.170196 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.170174 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.208631 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.208591 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3adc829c-gw-openshift-default-dockercfg-4gts2\"" Jul 28 18:27:47.253458 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.253413 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh"] Jul 28 18:27:47.260850 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.260812 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-credential-socket\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.260863 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-podinfo\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.260904 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-envoy\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.260928 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-data\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.261036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-token\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.261094 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl46v\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-kube-api-access-dl46v\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.261124 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-socket\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261301 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.261203 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istiod-ca-cert\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.261301 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.261248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-certs\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.330746 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.330713 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:47.362663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362600 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-token\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.362663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362701 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl46v\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-kube-api-access-dl46v\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.362663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362733 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-socket\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362781 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istiod-ca-cert\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362814 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-certs\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362894 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-credential-socket\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362919 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-podinfo\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362953 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-envoy\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.362975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-data\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.363201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-socket\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.363264 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-certs\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.363281 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-credential-socket\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363628 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.363608 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-data\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.363963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.363940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istiod-ca-cert\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.365367 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.365340 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-envoy\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.365567 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.365548 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-podinfo\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.374106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.374077 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-token\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.374641 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.374618 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl46v\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-kube-api-access-dl46v\") pod \"e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.480270 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.480163 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:47.539778 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.536477 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:47.626529 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.626499 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh"] Jul 28 18:27:47.627593 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:27:47.627566 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8445d7_fd5e_480d_89f6_e8f4289e36ce.slice/crio-08f5e78f9abd6536e0166b7418ad9117ec87fd477ae54c2187285f816e5eb645 WatchSource:0}: Error finding container 08f5e78f9abd6536e0166b7418ad9117ec87fd477ae54c2187285f816e5eb645: Status 404 returned error can't find the container with id 08f5e78f9abd6536e0166b7418ad9117ec87fd477ae54c2187285f816e5eb645 Jul 28 18:27:47.629795 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.629761 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:47.629852 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.629826 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:47.629892 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.629854 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:27:47.862944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.862837 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" event={"ID":"6b8445d7-fd5e-480d-89f6-e8f4289e36ce","Type":"ContainerStarted","Data":"d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957"} Jul 28 18:27:47.862944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.862902 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" event={"ID":"6b8445d7-fd5e-480d-89f6-e8f4289e36ce","Type":"ContainerStarted","Data":"08f5e78f9abd6536e0166b7418ad9117ec87fd477ae54c2187285f816e5eb645"} Jul 28 18:27:47.916925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:47.916865 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podStartSLOduration=0.916850055 podStartE2EDuration="916.850055ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:27:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:27:47.914634329 +0000 UTC m=+1865.329291332" watchObservedRunningTime="2026-07-28 18:27:47.916850055 +0000 UTC m=+1865.331507057" Jul 28 18:27:48.084331 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:48.084284 2575 patch_prober.go:28] interesting pod/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 28 18:27:48.084587 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:48.084359 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:27:48.480606 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:48.480563 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:48.485277 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:48.485250 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:48.869613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:48.869526 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:48.870543 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:48.870522 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:27:49.597401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:49.597365 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:49.743951 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:49.743919 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:49.969891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:49.969857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:50.492295 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:50.492265 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:50.611568 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:50.611535 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:27:51.256789 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.256756 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:51.402043 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402005 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8bsw\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-kube-api-access-m8bsw\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402056 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-certs\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402098 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istiod-ca-cert\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-data\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402173 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-envoy\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402316 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-token\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402371 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-podinfo\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402424 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-socket\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402454 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-credential-socket\") pod \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\" (UID: \"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd\") " Jul 28 18:27:51.402487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402458 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:27:51.402644 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402555 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:27:51.402745 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402716 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:27:51.402906 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402884 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.402996 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402909 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.402996 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402922 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.402996 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.402919 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:27:51.403144 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.403044 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-data" (OuterVolumeSpecName: "istio-data") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:27:51.404947 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.404924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:27:51.405029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.404949 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-token" (OuterVolumeSpecName: "istio-token") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:27:51.405029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.404991 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-kube-api-access-m8bsw" (OuterVolumeSpecName: "kube-api-access-m8bsw") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "kube-api-access-m8bsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:27:51.405029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.405011 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" (UID: "67ac8c6b-0d09-4db0-91bf-9fefee6e86fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:27:51.504170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.504127 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8bsw\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-kube-api-access-m8bsw\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.504170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.504161 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.504170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.504175 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.504417 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.504186 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.504417 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.504198 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.504417 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.504209 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:27:51.887745 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.887677 2575 generic.go:356] "Generic (PLEG): container finished" podID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerID="d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037" exitCode=0 Jul 28 18:27:51.887946 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.887748 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" Jul 28 18:27:51.887946 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.887773 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" event={"ID":"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd","Type":"ContainerDied","Data":"d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037"} Jul 28 18:27:51.887946 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.887825 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9" event={"ID":"67ac8c6b-0d09-4db0-91bf-9fefee6e86fd","Type":"ContainerDied","Data":"02079092656e4f7420aa056aa7add78b1291077b445c80365bd746cf3ab4da81"} Jul 28 18:27:51.887946 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.887848 2575 scope.go:117] "RemoveContainer" containerID="d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037" Jul 28 18:27:51.898646 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.898617 2575 scope.go:117] "RemoveContainer" containerID="4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684" Jul 28 18:27:51.909712 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.909676 2575 scope.go:117] "RemoveContainer" containerID="d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037" Jul 28 18:27:51.910059 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:27:51.910015 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037\": container with ID starting with d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037 not found: ID does not exist" containerID="d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037" Jul 28 18:27:51.910132 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.910067 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037"} err="failed to get container status \"d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037\": rpc error: code = NotFound desc = could not find container \"d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037\": container with ID starting with d48c989b6a8331d3453f4f21261139af16466de7c4a8f0f4351ede6a97325037 not found: ID does not exist" Jul 28 18:27:51.910132 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.910087 2575 scope.go:117] "RemoveContainer" containerID="4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684" Jul 28 18:27:51.910396 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:27:51.910379 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684\": container with ID starting with 4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684 not found: ID does not exist" containerID="4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684" Jul 28 18:27:51.910442 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.910402 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684"} err="failed to get container status \"4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684\": rpc error: code = NotFound desc = could not find container \"4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684\": container with ID starting with 4f79d32593ebc8e2590e83e1b63701c63805d1acfc6d9d28989415a731680684 not found: ID does not exist" Jul 28 18:27:51.918154 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.918122 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9"] Jul 28 18:27:51.924727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:51.924677 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0ab65c1c-openshift-default-7d9bb8c58-9sws9"] Jul 28 18:27:53.061976 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:27:53.061878 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" path="/var/lib/kubelet/pods/67ac8c6b-0d09-4db0-91bf-9fefee6e86fd/volumes" Jul 28 18:28:03.935382 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:03.935335 2575 patch_prober.go:28] interesting pod/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:47952->10.132.0.51:15021: read: connection reset by peer" start-of-body= Jul 28 18:28:03.935830 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:03.935414 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": read tcp 10.132.0.2:47952->10.132.0.51:15021: read: connection reset by peer" Jul 28 18:28:04.967948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:04.967868 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerID="d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957" exitCode=0 Jul 28 18:28:04.967948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:04.967940 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" event={"ID":"6b8445d7-fd5e-480d-89f6-e8f4289e36ce","Type":"ContainerDied","Data":"d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957"} Jul 28 18:28:04.968359 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:04.968331 2575 scope.go:117] "RemoveContainer" containerID="d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957" Jul 28 18:28:04.969385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:04.969349 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:04.969485 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:04.969446 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:04.969554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:04.969496 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:05.975172 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:05.975133 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" event={"ID":"6b8445d7-fd5e-480d-89f6-e8f4289e36ce","Type":"ContainerStarted","Data":"617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d"} Jul 28 18:28:06.481059 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:06.481012 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:28:06.481251 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:06.481233 2575 patch_prober.go:28] interesting pod/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 18:28:06.481319 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:06.481298 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 18:28:07.480873 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:07.480832 2575 patch_prober.go:28] interesting pod/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 18:28:07.481279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:07.480894 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 18:28:08.480739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:08.480666 2575 patch_prober.go:28] interesting pod/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 18:28:08.480924 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:08.480775 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 18:28:09.481425 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:09.481386 2575 patch_prober.go:28] interesting pod/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Jul 28 18:28:09.481886 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:09.481447 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Jul 28 18:28:10.484956 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:10.484920 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:28:10.485365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:10.485262 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:28:10.486096 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:10.486078 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:28:14.627453 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.627411 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk"] Jul 28 18:28:14.627907 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.627836 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" Jul 28 18:28:14.627907 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.627848 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" Jul 28 18:28:14.627907 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.627867 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" Jul 28 18:28:14.627907 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.627872 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" Jul 28 18:28:14.628061 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.627928 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" Jul 28 18:28:14.630549 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.630531 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:14.633299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.633276 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:28:14.633416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.633313 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:28:14.639357 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.639334 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk"] Jul 28 18:28:14.715716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.715633 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b905c761-0705-4152-aa42-1dc8578627d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk\" (UID: \"b905c761-0705-4152-aa42-1dc8578627d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:14.816517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.816479 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b905c761-0705-4152-aa42-1dc8578627d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk\" (UID: \"b905c761-0705-4152-aa42-1dc8578627d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:14.817163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.817144 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b905c761-0705-4152-aa42-1dc8578627d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk\" (UID: \"b905c761-0705-4152-aa42-1dc8578627d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:14.941707 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:14.941643 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:15.075503 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:15.075479 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk"] Jul 28 18:28:15.077094 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:28:15.077065 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb905c761_0705_4152_aa42_1dc8578627d2.slice/crio-31c24f307cbb3cedd7b16cfe44eafad6daa81aef4469c657d5f17dc762fd50d9 WatchSource:0}: Error finding container 31c24f307cbb3cedd7b16cfe44eafad6daa81aef4469c657d5f17dc762fd50d9: Status 404 returned error can't find the container with id 31c24f307cbb3cedd7b16cfe44eafad6daa81aef4469c657d5f17dc762fd50d9 Jul 28 18:28:16.031541 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:16.031434 2575 generic.go:356] "Generic (PLEG): container finished" podID="b905c761-0705-4152-aa42-1dc8578627d2" containerID="9f734b2081da1748963026f711015e4112d28da5862b9de58eb91e0b893892fe" exitCode=0 Jul 28 18:28:16.031951 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:16.031531 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" event={"ID":"b905c761-0705-4152-aa42-1dc8578627d2","Type":"ContainerDied","Data":"9f734b2081da1748963026f711015e4112d28da5862b9de58eb91e0b893892fe"} Jul 28 18:28:16.031951 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:16.031587 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" event={"ID":"b905c761-0705-4152-aa42-1dc8578627d2","Type":"ContainerStarted","Data":"31c24f307cbb3cedd7b16cfe44eafad6daa81aef4469c657d5f17dc762fd50d9"} Jul 28 18:28:17.172018 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:17.171993 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:17.238276 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:17.238241 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b905c761-0705-4152-aa42-1dc8578627d2-maas-api-service-ca\") pod \"b905c761-0705-4152-aa42-1dc8578627d2\" (UID: \"b905c761-0705-4152-aa42-1dc8578627d2\") " Jul 28 18:28:17.238602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:17.238579 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b905c761-0705-4152-aa42-1dc8578627d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b905c761-0705-4152-aa42-1dc8578627d2" (UID: "b905c761-0705-4152-aa42-1dc8578627d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:28:17.339339 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:17.339250 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b905c761-0705-4152-aa42-1dc8578627d2-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:18.044328 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:18.044298 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" Jul 28 18:28:18.044502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:18.044294 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk" event={"ID":"b905c761-0705-4152-aa42-1dc8578627d2","Type":"ContainerDied","Data":"31c24f307cbb3cedd7b16cfe44eafad6daa81aef4469c657d5f17dc762fd50d9"} Jul 28 18:28:18.044502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:18.044456 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31c24f307cbb3cedd7b16cfe44eafad6daa81aef4469c657d5f17dc762fd50d9" Jul 28 18:28:20.402953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.402918 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:20.499931 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.499896 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:20.600772 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.600736 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:20.656952 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.656857 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk"] Jul 28 18:28:20.659039 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.659009 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3adc829c-wkfqk"] Jul 28 18:28:20.698594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.698562 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:20.803099 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.803066 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh"] Jul 28 18:28:20.803289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.803263 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" containerID="cri-o://617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d" gracePeriod=30 Jul 28 18:28:20.829425 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:20.829391 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:21.052779 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.052636 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:21.058730 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.058700 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b905c761-0705-4152-aa42-1dc8578627d2" path="/var/lib/kubelet/pods/b905c761-0705-4152-aa42-1dc8578627d2/volumes" Jul 28 18:28:21.296451 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.296409 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:21.622088 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.622050 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv"] Jul 28 18:28:21.622483 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.622468 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b905c761-0705-4152-aa42-1dc8578627d2" containerName="revoke-keys" Jul 28 18:28:21.622527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.622485 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b905c761-0705-4152-aa42-1dc8578627d2" containerName="revoke-keys" Jul 28 18:28:21.622562 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.622554 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b905c761-0705-4152-aa42-1dc8578627d2" containerName="revoke-keys" Jul 28 18:28:21.622601 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.622564 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="67ac8c6b-0d09-4db0-91bf-9fefee6e86fd" containerName="istio-proxy" Jul 28 18:28:21.625807 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.625789 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.631036 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.631011 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-223d27c2-openshift-default-dockercfg-kj2kd\"" Jul 28 18:28:21.638149 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.638125 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:21.653101 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.653065 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv"] Jul 28 18:28:21.782368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782328 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-token\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782379 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69csj\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-kube-api-access-69csj\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782458 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-socket\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782505 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-credential-socket\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782580 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-envoy\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782651 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-certs\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782768 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-podinfo\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782935 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782811 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-data\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.782935 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.782836 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc347066-3dd6-43bf-ae68-ee108ba86351-istiod-ca-cert\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884127 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884092 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-envoy\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884152 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-certs\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884171 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-podinfo\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884194 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-data\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884211 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc347066-3dd6-43bf-ae68-ee108ba86351-istiod-ca-cert\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884229 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-token\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884275 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884249 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69csj\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-kube-api-access-69csj\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884294 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-socket\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884322 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-credential-socket\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884813 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-credential-socket\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.884932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884917 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-data\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.885095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.884995 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-certs\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.885095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.885045 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc347066-3dd6-43bf-ae68-ee108ba86351-istiod-ca-cert\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.885172 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.885109 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-socket\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.886547 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.886523 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-envoy\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.887055 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.887033 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-podinfo\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.894135 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.894113 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69csj\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-kube-api-access-69csj\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.894258 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.894112 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-token\") pod \"e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:21.936324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:21.936283 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:22.092765 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:22.092737 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv"] Jul 28 18:28:22.094271 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:28:22.094238 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc347066_3dd6_43bf_ae68_ee108ba86351.slice/crio-5f10797c0ce652331738c95661ddcfa3e03ed497f0a7e6878a71940584b5e5e7 WatchSource:0}: Error finding container 5f10797c0ce652331738c95661ddcfa3e03ed497f0a7e6878a71940584b5e5e7: Status 404 returned error can't find the container with id 5f10797c0ce652331738c95661ddcfa3e03ed497f0a7e6878a71940584b5e5e7 Jul 28 18:28:22.096472 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:22.096436 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:22.096571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:22.096505 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:22.096571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:22.096534 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:22.412695 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:22.412585 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:23.078663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:23.078629 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" event={"ID":"fc347066-3dd6-43bf-ae68-ee108ba86351","Type":"ContainerStarted","Data":"3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa"} Jul 28 18:28:23.079120 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:23.078706 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" event={"ID":"fc347066-3dd6-43bf-ae68-ee108ba86351","Type":"ContainerStarted","Data":"5f10797c0ce652331738c95661ddcfa3e03ed497f0a7e6878a71940584b5e5e7"} Jul 28 18:28:23.104882 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:23.104820 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podStartSLOduration=2.104799934 podStartE2EDuration="2.104799934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:23.102487354 +0000 UTC m=+1900.517144356" watchObservedRunningTime="2026-07-28 18:28:23.104799934 +0000 UTC m=+1900.519456936" Jul 28 18:28:23.936967 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:23.936923 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:23.941764 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:23.941729 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:24.084735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:24.084700 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:24.085803 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:24.085780 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:24.449061 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:24.449026 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:24.549500 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:24.549464 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:24.673161 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:24.673124 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:25.039623 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:25.039580 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:25.360166 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:25.360079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:25.461753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:25.461711 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:25.487666 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:25.487626 2575 patch_prober.go:28] interesting pod/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:28:25.487888 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:25.487734 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:28:26.065442 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.065362 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:28:26.099490 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.099455 2575 generic.go:356] "Generic (PLEG): container finished" podID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerID="617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d" exitCode=0 Jul 28 18:28:26.099678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.099512 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" Jul 28 18:28:26.099678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.099541 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" event={"ID":"6b8445d7-fd5e-480d-89f6-e8f4289e36ce","Type":"ContainerDied","Data":"617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d"} Jul 28 18:28:26.099678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.099608 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh" event={"ID":"6b8445d7-fd5e-480d-89f6-e8f4289e36ce","Type":"ContainerDied","Data":"08f5e78f9abd6536e0166b7418ad9117ec87fd477ae54c2187285f816e5eb645"} Jul 28 18:28:26.099678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.099638 2575 scope.go:117] "RemoveContainer" containerID="617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d" Jul 28 18:28:26.109295 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.109275 2575 scope.go:117] "RemoveContainer" containerID="d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957" Jul 28 18:28:26.118858 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.118837 2575 scope.go:117] "RemoveContainer" containerID="617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d" Jul 28 18:28:26.119205 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:28:26.119177 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d\": container with ID starting with 617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d not found: ID does not exist" containerID="617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d" Jul 28 18:28:26.119314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.119207 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d"} err="failed to get container status \"617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d\": rpc error: code = NotFound desc = could not find container \"617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d\": container with ID starting with 617170dd4fcf9cf14fcecb8ad950dc41be52b46e1647158ddf3d359217002c4d not found: ID does not exist" Jul 28 18:28:26.119314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.119227 2575 scope.go:117] "RemoveContainer" containerID="d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957" Jul 28 18:28:26.119496 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:28:26.119477 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957\": container with ID starting with d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957 not found: ID does not exist" containerID="d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957" Jul 28 18:28:26.119539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.119503 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957"} err="failed to get container status \"d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957\": rpc error: code = NotFound desc = could not find container \"d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957\": container with ID starting with d0fdf5aefb9168993b0340f3049e61c91002534669d40376a0abf4d9c90e9957 not found: ID does not exist" Jul 28 18:28:26.225050 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225009 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-podinfo\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225050 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225051 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-token\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225069 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-certs\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225098 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dl46v\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-kube-api-access-dl46v\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-credential-socket\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225142 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-envoy\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225158 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-socket\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225180 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istiod-ca-cert\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225244 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-data\") pod \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\" (UID: \"6b8445d7-fd5e-480d-89f6-e8f4289e36ce\") " Jul 28 18:28:26.225739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225429 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:26.225739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225478 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:26.225739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225562 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:26.225978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225955 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.226039 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.225986 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.226039 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.226002 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.226039 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.226002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:28:26.226751 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.226722 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:26.227531 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.227501 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:28:26.227531 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.227514 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:28:26.227737 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.227575 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-kube-api-access-dl46v" (OuterVolumeSpecName: "kube-api-access-dl46v") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "kube-api-access-dl46v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:28:26.227881 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.227864 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b8445d7-fd5e-480d-89f6-e8f4289e36ce" (UID: "6b8445d7-fd5e-480d-89f6-e8f4289e36ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:28:26.327156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.327064 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.327156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.327098 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.327156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.327108 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl46v\" (UniqueName: \"kubernetes.io/projected/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-kube-api-access-dl46v\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.327156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.327120 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.327156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.327130 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.327156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.327138 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b8445d7-fd5e-480d-89f6-e8f4289e36ce-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:26.424410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.424375 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh"] Jul 28 18:28:26.428938 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:26.428908 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3adc829c-gw-openshift-default-5469d4d888-6scxh"] Jul 28 18:28:27.060041 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:27.060003 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" path="/var/lib/kubelet/pods/6b8445d7-fd5e-480d-89f6-e8f4289e36ce/volumes" Jul 28 18:28:39.086197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.086155 2575 patch_prober.go:28] interesting pod/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 18:28:39.086632 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.086213 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 18:28:39.187862 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.187827 2575 generic.go:356] "Generic (PLEG): container finished" podID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerID="3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa" exitCode=0 Jul 28 18:28:39.188047 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.187903 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" event={"ID":"fc347066-3dd6-43bf-ae68-ee108ba86351","Type":"ContainerDied","Data":"3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa"} Jul 28 18:28:39.188324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.188300 2575 scope.go:117] "RemoveContainer" containerID="3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa" Jul 28 18:28:39.189352 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.189318 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:39.189428 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.189409 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:39.189473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:39.189454 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:40.195277 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:40.195233 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" event={"ID":"fc347066-3dd6-43bf-ae68-ee108ba86351","Type":"ContainerStarted","Data":"8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7"} Jul 28 18:28:40.937857 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:40.937819 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:40.938311 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:40.938223 2575 patch_prober.go:28] interesting pod/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 18:28:40.938492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:40.938469 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 18:28:41.937235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:41.937192 2575 patch_prober.go:28] interesting pod/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 18:28:41.938288 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:41.937850 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 18:28:42.936838 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:42.936787 2575 patch_prober.go:28] interesting pod/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 18:28:42.937188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:42.937137 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 18:28:43.937077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:43.937024 2575 patch_prober.go:28] interesting pod/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 28 18:28:43.937602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:43.937103 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 28 18:28:44.941146 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:44.941110 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:44.941631 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:44.941381 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:44.942221 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:44.942205 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:28:50.320087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320052 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx"] Jul 28 18:28:50.320551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320434 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" Jul 28 18:28:50.320551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320445 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" Jul 28 18:28:50.320551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320457 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" Jul 28 18:28:50.320551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320463 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" Jul 28 18:28:50.320551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320518 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" Jul 28 18:28:50.320551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.320526 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8445d7-fd5e-480d-89f6-e8f4289e36ce" containerName="istio-proxy" Jul 28 18:28:50.322934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.322917 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:50.325474 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.325448 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:28:50.325623 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.325474 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:28:50.332947 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.332915 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx"] Jul 28 18:28:50.347553 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.347514 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50622dff-8601-47c1-9060-29bf2b7ed991-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-223d27c2-v59jx\" (UID: \"50622dff-8601-47c1-9060-29bf2b7ed991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:50.448947 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.448904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50622dff-8601-47c1-9060-29bf2b7ed991-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-223d27c2-v59jx\" (UID: \"50622dff-8601-47c1-9060-29bf2b7ed991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:50.449590 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.449562 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50622dff-8601-47c1-9060-29bf2b7ed991-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-223d27c2-v59jx\" (UID: \"50622dff-8601-47c1-9060-29bf2b7ed991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:50.634539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.634505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:50.776049 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:50.775990 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx"] Jul 28 18:28:50.779114 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:28:50.779084 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50622dff_8601_47c1_9060_29bf2b7ed991.slice/crio-5588967e7725c402d8d5443819a2a9977a9bfce94c0c98481428496567c0d010 WatchSource:0}: Error finding container 5588967e7725c402d8d5443819a2a9977a9bfce94c0c98481428496567c0d010: Status 404 returned error can't find the container with id 5588967e7725c402d8d5443819a2a9977a9bfce94c0c98481428496567c0d010 Jul 28 18:28:51.259144 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:51.259042 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" event={"ID":"50622dff-8601-47c1-9060-29bf2b7ed991","Type":"ContainerStarted","Data":"bcb4c1e68356e7720b6b695de671fd8c3cfad5200d7d5e4ed160efb33479b4c4"} Jul 28 18:28:51.259144 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:51.259098 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" event={"ID":"50622dff-8601-47c1-9060-29bf2b7ed991","Type":"ContainerStarted","Data":"5588967e7725c402d8d5443819a2a9977a9bfce94c0c98481428496567c0d010"} Jul 28 18:28:51.280829 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:51.280775 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" podStartSLOduration=1.280759517 podStartE2EDuration="1.280759517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:51.279834372 +0000 UTC m=+1928.694491373" watchObservedRunningTime="2026-07-28 18:28:51.280759517 +0000 UTC m=+1928.695416522" Jul 28 18:28:52.265855 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:52.265815 2575 generic.go:356] "Generic (PLEG): container finished" podID="50622dff-8601-47c1-9060-29bf2b7ed991" containerID="bcb4c1e68356e7720b6b695de671fd8c3cfad5200d7d5e4ed160efb33479b4c4" exitCode=0 Jul 28 18:28:52.266255 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:52.265894 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" event={"ID":"50622dff-8601-47c1-9060-29bf2b7ed991","Type":"ContainerDied","Data":"bcb4c1e68356e7720b6b695de671fd8c3cfad5200d7d5e4ed160efb33479b4c4"} Jul 28 18:28:53.410461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:53.410429 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:53.475649 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:53.475605 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50622dff-8601-47c1-9060-29bf2b7ed991-maas-api-service-ca\") pod \"50622dff-8601-47c1-9060-29bf2b7ed991\" (UID: \"50622dff-8601-47c1-9060-29bf2b7ed991\") " Jul 28 18:28:53.476041 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:53.476014 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50622dff-8601-47c1-9060-29bf2b7ed991-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "50622dff-8601-47c1-9060-29bf2b7ed991" (UID: "50622dff-8601-47c1-9060-29bf2b7ed991"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:28:53.577189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:53.577096 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/50622dff-8601-47c1-9060-29bf2b7ed991-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:28:53.974604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:53.974567 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:54.278517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:54.278421 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" Jul 28 18:28:54.278517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:54.278461 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx" event={"ID":"50622dff-8601-47c1-9060-29bf2b7ed991","Type":"ContainerDied","Data":"5588967e7725c402d8d5443819a2a9977a9bfce94c0c98481428496567c0d010"} Jul 28 18:28:54.278517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:54.278508 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5588967e7725c402d8d5443819a2a9977a9bfce94c0c98481428496567c0d010" Jul 28 18:28:55.403516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:55.403482 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:55.483202 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:55.483164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:55.559967 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:55.559914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:55.579848 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:55.579797 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx"] Jul 28 18:28:55.581997 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:55.581972 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223d27c2-v59jx"] Jul 28 18:28:55.628266 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:55.628226 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:56.855281 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:56.855246 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv"] Jul 28 18:28:56.855783 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:56.855460 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" containerID="cri-o://8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7" gracePeriod=30 Jul 28 18:28:56.865186 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:56.865156 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:57.059270 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.059231 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50622dff-8601-47c1-9060-29bf2b7ed991" path="/var/lib/kubelet/pods/50622dff-8601-47c1-9060-29bf2b7ed991/volumes" Jul 28 18:28:57.063609 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.063584 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:57.297323 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.297290 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:57.619418 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.619327 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r"] Jul 28 18:28:57.619773 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.619757 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50622dff-8601-47c1-9060-29bf2b7ed991" containerName="revoke-keys" Jul 28 18:28:57.619847 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.619777 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="50622dff-8601-47c1-9060-29bf2b7ed991" containerName="revoke-keys" Jul 28 18:28:57.619884 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.619879 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="50622dff-8601-47c1-9060-29bf2b7ed991" containerName="revoke-keys" Jul 28 18:28:57.622146 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.622127 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.625066 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.625038 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-49ca23b2-openshift-default-dockercfg-nggsp\"" Jul 28 18:28:57.640269 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.640230 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r"] Jul 28 18:28:57.659918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.659880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:57.714394 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714358 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-envoy\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714394 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714393 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-istio-token\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714421 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-credential-socket\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714475 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-data\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714513 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58t4q\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-kube-api-access-58t4q\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714572 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5936f75f-1443-41ed-b83b-0dd86394195a-istio-podinfo\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714594 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5936f75f-1443-41ed-b83b-0dd86394195a-istiod-ca-cert\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-socket\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.714671 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.714658 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-certs\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.772476 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.772445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:57.815483 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815445 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-envoy\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.815483 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-istio-token\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.815757 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815506 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-credential-socket\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.815757 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815529 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-data\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.815757 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815642 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58t4q\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-kube-api-access-58t4q\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.815757 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5936f75f-1443-41ed-b83b-0dd86394195a-istio-podinfo\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815787 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5936f75f-1443-41ed-b83b-0dd86394195a-istiod-ca-cert\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815906 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-socket\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.815937 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-certs\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.816012 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-credential-socket\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816245 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.816035 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-data\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816303 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.816268 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-socket\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816303 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.816285 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-certs\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.816761 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.816744 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5936f75f-1443-41ed-b83b-0dd86394195a-istiod-ca-cert\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.818093 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.818071 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-envoy\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.818659 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.818634 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5936f75f-1443-41ed-b83b-0dd86394195a-istio-podinfo\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.830539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.830504 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-istio-token\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.831402 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.831380 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58t4q\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-kube-api-access-58t4q\") pod \"e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:57.933632 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:57.933589 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:58.083594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.083563 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r"] Jul 28 18:28:58.085282 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:28:58.085246 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5936f75f_1443_41ed_b83b_0dd86394195a.slice/crio-eb3a86e195d26d214ea3ba1730cacb3e21ffe3886339515efa00ff49957b6df0 WatchSource:0}: Error finding container eb3a86e195d26d214ea3ba1730cacb3e21ffe3886339515efa00ff49957b6df0: Status 404 returned error can't find the container with id eb3a86e195d26d214ea3ba1730cacb3e21ffe3886339515efa00ff49957b6df0 Jul 28 18:28:58.087282 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.087248 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:58.087356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.087314 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:58.087356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.087341 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:28:58.305841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.305722 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" event={"ID":"5936f75f-1443-41ed-b83b-0dd86394195a","Type":"ContainerStarted","Data":"0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6"} Jul 28 18:28:58.306024 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.305838 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" event={"ID":"5936f75f-1443-41ed-b83b-0dd86394195a","Type":"ContainerStarted","Data":"eb3a86e195d26d214ea3ba1730cacb3e21ffe3886339515efa00ff49957b6df0"} Jul 28 18:28:58.326473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.326411 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podStartSLOduration=1.326390467 podStartE2EDuration="1.326390467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:28:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:28:58.325056663 +0000 UTC m=+1935.739713665" watchObservedRunningTime="2026-07-28 18:28:58.326390467 +0000 UTC m=+1935.741047470" Jul 28 18:28:58.934464 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:58.934426 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:59.768838 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:59.768799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:59.896551 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:59.896516 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:28:59.940324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:59.940289 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:28:59.944244 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:59.944207 2575 patch_prober.go:28] interesting pod/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:28:59.944401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:28:59.944266 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:29:00.014991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:00.014956 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:00.151064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:00.151028 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:00.318157 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:00.318123 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:00.319152 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:00.319126 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:00.612235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:00.612115 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:02.115048 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.114978 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:29:02.257176 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257133 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-socket\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257186 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-data\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257230 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-envoy\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257262 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-podinfo\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257289 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-credential-socket\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257356 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-token\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257390 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc347066-3dd6-43bf-ae68-ee108ba86351-istiod-ca-cert\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257736 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257435 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-certs\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257736 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257500 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69csj\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-kube-api-access-69csj\") pod \"fc347066-3dd6-43bf-ae68-ee108ba86351\" (UID: \"fc347066-3dd6-43bf-ae68-ee108ba86351\") " Jul 28 18:29:02.257736 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257512 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:02.257736 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257654 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:02.257937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257832 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.257937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257848 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc347066-3dd6-43bf-ae68-ee108ba86351-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:29:02.257937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257852 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.257937 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.257849 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:02.258315 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.258294 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-data" (OuterVolumeSpecName: "istio-data") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:02.259613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.259588 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:02.259613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.259595 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:29:02.260026 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.260002 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-token" (OuterVolumeSpecName: "istio-token") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:29:02.260089 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.260045 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-kube-api-access-69csj" (OuterVolumeSpecName: "kube-api-access-69csj") pod "fc347066-3dd6-43bf-ae68-ee108ba86351" (UID: "fc347066-3dd6-43bf-ae68-ee108ba86351"). InnerVolumeSpecName "kube-api-access-69csj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:29:02.331895 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.331860 2575 generic.go:356] "Generic (PLEG): container finished" podID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerID="8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7" exitCode=0 Jul 28 18:29:02.332059 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.331910 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" Jul 28 18:29:02.332059 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.331946 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" event={"ID":"fc347066-3dd6-43bf-ae68-ee108ba86351","Type":"ContainerDied","Data":"8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7"} Jul 28 18:29:02.332059 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.332001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv" event={"ID":"fc347066-3dd6-43bf-ae68-ee108ba86351","Type":"ContainerDied","Data":"5f10797c0ce652331738c95661ddcfa3e03ed497f0a7e6878a71940584b5e5e7"} Jul 28 18:29:02.332059 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.332024 2575 scope.go:117] "RemoveContainer" containerID="8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7" Jul 28 18:29:02.342335 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.342316 2575 scope.go:117] "RemoveContainer" containerID="3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa" Jul 28 18:29:02.351642 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.351619 2575 scope.go:117] "RemoveContainer" containerID="8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7" Jul 28 18:29:02.352137 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:29:02.352103 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7\": container with ID starting with 8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7 not found: ID does not exist" containerID="8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7" Jul 28 18:29:02.352262 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.352148 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7"} err="failed to get container status \"8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7\": rpc error: code = NotFound desc = could not find container \"8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7\": container with ID starting with 8068589a7af60b5402fedd35e7d11d38187d749d90e31e3dcc1c59b91fd25de7 not found: ID does not exist" Jul 28 18:29:02.352262 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.352173 2575 scope.go:117] "RemoveContainer" containerID="3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa" Jul 28 18:29:02.353079 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:29:02.353028 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa\": container with ID starting with 3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa not found: ID does not exist" containerID="3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa" Jul 28 18:29:02.353177 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.353087 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa"} err="failed to get container status \"3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa\": rpc error: code = NotFound desc = could not find container \"3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa\": container with ID starting with 3fa4cfed7a5410da34e648edfe052b2dbfc028587ffe65a2b03da49c94971cfa not found: ID does not exist" Jul 28 18:29:02.354930 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.354910 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv"] Jul 28 18:29:02.358988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.358963 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.358988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.358989 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc347066-3dd6-43bf-ae68-ee108ba86351-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.359163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.359004 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.359163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.359016 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69csj\" (UniqueName: \"kubernetes.io/projected/fc347066-3dd6-43bf-ae68-ee108ba86351-kube-api-access-69csj\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.359163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.359029 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.359163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.359041 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.359163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.359051 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc347066-3dd6-43bf-ae68-ee108ba86351-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:02.360074 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.360053 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-223d27c2-openshift-default-6f89b99f86-svkfv"] Jul 28 18:29:02.486339 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:02.486304 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:03.059839 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:03.059798 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" path="/var/lib/kubelet/pods/fc347066-3dd6-43bf-ae68-ee108ba86351/volumes" Jul 28 18:29:13.399371 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:13.399326 2575 generic.go:356] "Generic (PLEG): container finished" podID="5936f75f-1443-41ed-b83b-0dd86394195a" containerID="0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6" exitCode=0 Jul 28 18:29:13.399948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:13.399561 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" event={"ID":"5936f75f-1443-41ed-b83b-0dd86394195a","Type":"ContainerDied","Data":"0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6"} Jul 28 18:29:13.400282 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:13.400259 2575 scope.go:117] "RemoveContainer" containerID="0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6" Jul 28 18:29:13.402490 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:13.402439 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:29:13.402630 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:13.402600 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:29:13.402735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:13.402716 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:29:14.406930 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:14.406884 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" event={"ID":"5936f75f-1443-41ed-b83b-0dd86394195a","Type":"ContainerStarted","Data":"5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8"} Jul 28 18:29:14.935232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:14.935189 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:14.935731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:14.935671 2575 patch_prober.go:28] interesting pod/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 18:29:14.935863 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:14.935758 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 18:29:15.318943 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:15.318841 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:15.935075 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:15.935032 2575 patch_prober.go:28] interesting pod/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 18:29:15.935590 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:15.935097 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 18:29:16.934227 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:16.934181 2575 patch_prober.go:28] interesting pod/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 18:29:16.934454 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:16.934251 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 18:29:17.934259 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:17.934214 2575 patch_prober.go:28] interesting pod/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 28 18:29:17.934798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:17.934290 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 28 18:29:18.938168 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:18.938136 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:18.939353 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:18.939328 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:24.694063 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.694019 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm"] Jul 28 18:29:24.694533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.694408 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" Jul 28 18:29:24.694533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.694419 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" Jul 28 18:29:24.694533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.694432 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" Jul 28 18:29:24.694533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.694441 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" Jul 28 18:29:24.694533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.694504 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" Jul 28 18:29:24.697600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.697580 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:24.700407 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.700382 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:29:24.700558 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.700390 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:29:24.705258 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.705219 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm"] Jul 28 18:29:24.870885 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.870846 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8f4153-d59a-4423-85ca-eb18263169f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm\" (UID: \"3d8f4153-d59a-4423-85ca-eb18263169f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:24.972309 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.972191 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8f4153-d59a-4423-85ca-eb18263169f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm\" (UID: \"3d8f4153-d59a-4423-85ca-eb18263169f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:24.972887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:24.972866 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8f4153-d59a-4423-85ca-eb18263169f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm\" (UID: \"3d8f4153-d59a-4423-85ca-eb18263169f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:25.012259 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:25.012211 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:25.165051 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:25.165014 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm"] Jul 28 18:29:25.475351 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:25.475314 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" event={"ID":"3d8f4153-d59a-4423-85ca-eb18263169f5","Type":"ContainerStarted","Data":"9967fda620b3d5aaeea106678c528391dae65797b9b88272d7f9aac613d7ae36"} Jul 28 18:29:25.475534 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:25.475375 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" event={"ID":"3d8f4153-d59a-4423-85ca-eb18263169f5","Type":"ContainerStarted","Data":"af75e3a6b1921b4cd521a4a812462ba247588c88d2377becce02ada6bf294844"} Jul 28 18:29:25.491926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:25.491868 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" podStartSLOduration=1.491851269 podStartE2EDuration="1.491851269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:29:25.489468351 +0000 UTC m=+1962.904125352" watchObservedRunningTime="2026-07-28 18:29:25.491851269 +0000 UTC m=+1962.906508271" Jul 28 18:29:26.483497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:26.483461 2575 generic.go:356] "Generic (PLEG): container finished" podID="3d8f4153-d59a-4423-85ca-eb18263169f5" containerID="9967fda620b3d5aaeea106678c528391dae65797b9b88272d7f9aac613d7ae36" exitCode=0 Jul 28 18:29:26.483941 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:26.483518 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" event={"ID":"3d8f4153-d59a-4423-85ca-eb18263169f5","Type":"ContainerDied","Data":"9967fda620b3d5aaeea106678c528391dae65797b9b88272d7f9aac613d7ae36"} Jul 28 18:29:27.620022 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:27.619987 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:27.798055 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:27.797956 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8f4153-d59a-4423-85ca-eb18263169f5-maas-api-service-ca\") pod \"3d8f4153-d59a-4423-85ca-eb18263169f5\" (UID: \"3d8f4153-d59a-4423-85ca-eb18263169f5\") " Jul 28 18:29:27.798344 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:27.798316 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8f4153-d59a-4423-85ca-eb18263169f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d8f4153-d59a-4423-85ca-eb18263169f5" (UID: "3d8f4153-d59a-4423-85ca-eb18263169f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:29:27.899592 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:27.899550 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8f4153-d59a-4423-85ca-eb18263169f5-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:28.496011 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:28.495978 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" Jul 28 18:29:28.496192 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:28.496012 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm" event={"ID":"3d8f4153-d59a-4423-85ca-eb18263169f5","Type":"ContainerDied","Data":"af75e3a6b1921b4cd521a4a812462ba247588c88d2377becce02ada6bf294844"} Jul 28 18:29:28.496192 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:28.496090 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af75e3a6b1921b4cd521a4a812462ba247588c88d2377becce02ada6bf294844" Jul 28 18:29:30.691422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.691390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:30.794358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.794320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:30.859569 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.859518 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm"] Jul 28 18:29:30.861816 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.861789 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-49ca23b2-6mscm"] Jul 28 18:29:30.876344 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.876314 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:30.997873 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.997788 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r"] Jul 28 18:29:30.998039 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:30.998013 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" containerID="cri-o://5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8" gracePeriod=30 Jul 28 18:29:31.029711 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:31.029661 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:31.060264 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:31.060224 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d8f4153-d59a-4423-85ca-eb18263169f5" path="/var/lib/kubelet/pods/3d8f4153-d59a-4423-85ca-eb18263169f5/volumes" Jul 28 18:29:31.252866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:31.252774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:31.785918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:31.785884 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:31.886487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:31.886444 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:33.409857 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:33.409821 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:33.940901 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:33.940864 2575 patch_prober.go:28] interesting pod/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:29:33.941081 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:33.940929 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:29:36.262322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.262251 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:36.369463 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369423 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-envoy\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369463 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369470 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-credential-socket\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369766 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369521 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5936f75f-1443-41ed-b83b-0dd86394195a-istiod-ca-cert\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369766 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369594 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-istio-token\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369766 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369625 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58t4q\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-kube-api-access-58t4q\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369766 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369708 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-data\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5936f75f-1443-41ed-b83b-0dd86394195a-istio-podinfo\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369899 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-socket\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369923 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-certs\") pod \"5936f75f-1443-41ed-b83b-0dd86394195a\" (UID: \"5936f75f-1443-41ed-b83b-0dd86394195a\") " Jul 28 18:29:36.369978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.369946 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:36.370179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370001 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5936f75f-1443-41ed-b83b-0dd86394195a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:29:36.370179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370158 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:36.370410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370333 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5936f75f-1443-41ed-b83b-0dd86394195a-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.370410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370357 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.370410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370361 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:36.370410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370372 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.370725 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.370657 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-data" (OuterVolumeSpecName: "istio-data") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:36.371988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.371951 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-istio-token" (OuterVolumeSpecName: "istio-token") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:29:36.372094 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.372025 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5936f75f-1443-41ed-b83b-0dd86394195a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:29:36.372207 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.372190 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:29:36.372267 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.372246 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-kube-api-access-58t4q" (OuterVolumeSpecName: "kube-api-access-58t4q") pod "5936f75f-1443-41ed-b83b-0dd86394195a" (UID: "5936f75f-1443-41ed-b83b-0dd86394195a"). InnerVolumeSpecName "kube-api-access-58t4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:29:36.471035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.470985 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.471035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.471029 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58t4q\" (UniqueName: \"kubernetes.io/projected/5936f75f-1443-41ed-b83b-0dd86394195a-kube-api-access-58t4q\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.471035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.471041 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.471035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.471049 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5936f75f-1443-41ed-b83b-0dd86394195a-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.471035 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.471057 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.471348 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.471065 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5936f75f-1443-41ed-b83b-0dd86394195a-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:29:36.545072 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.544978 2575 generic.go:356] "Generic (PLEG): container finished" podID="5936f75f-1443-41ed-b83b-0dd86394195a" containerID="5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8" exitCode=0 Jul 28 18:29:36.545072 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.545035 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" Jul 28 18:29:36.545236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.545065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" event={"ID":"5936f75f-1443-41ed-b83b-0dd86394195a","Type":"ContainerDied","Data":"5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8"} Jul 28 18:29:36.545236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.545124 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r" event={"ID":"5936f75f-1443-41ed-b83b-0dd86394195a","Type":"ContainerDied","Data":"eb3a86e195d26d214ea3ba1730cacb3e21ffe3886339515efa00ff49957b6df0"} Jul 28 18:29:36.545236 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.545147 2575 scope.go:117] "RemoveContainer" containerID="5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8" Jul 28 18:29:36.555823 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.555798 2575 scope.go:117] "RemoveContainer" containerID="0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6" Jul 28 18:29:36.564897 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.564874 2575 scope.go:117] "RemoveContainer" containerID="5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8" Jul 28 18:29:36.565248 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:29:36.565219 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8\": container with ID starting with 5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8 not found: ID does not exist" containerID="5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8" Jul 28 18:29:36.565340 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.565262 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8"} err="failed to get container status \"5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8\": rpc error: code = NotFound desc = could not find container \"5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8\": container with ID starting with 5eb87bcf932ce916e93f21cf11640abf9b266d8699dd2ab069a3b815318910d8 not found: ID does not exist" Jul 28 18:29:36.565340 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.565289 2575 scope.go:117] "RemoveContainer" containerID="0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6" Jul 28 18:29:36.565697 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:29:36.565640 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6\": container with ID starting with 0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6 not found: ID does not exist" containerID="0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6" Jul 28 18:29:36.565816 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.565711 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6"} err="failed to get container status \"0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6\": rpc error: code = NotFound desc = could not find container \"0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6\": container with ID starting with 0e9d01ec5a780b0f664a972a989097483d18707b5301d1f16ed85219de2bf1b6 not found: ID does not exist" Jul 28 18:29:36.568460 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.568437 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r"] Jul 28 18:29:36.572425 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:36.572397 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-49ca23b2-openshift-default-8c6c4c59b-p4p2r"] Jul 28 18:29:37.060521 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:37.060482 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" path="/var/lib/kubelet/pods/5936f75f-1443-41ed-b83b-0dd86394195a/volumes" Jul 28 18:29:40.020598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:40.020562 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:40.129316 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:40.129281 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:40.256462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:40.256415 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:40.347452 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:40.347361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:29:40.819637 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:29:40.819604 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:00.135783 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.135741 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8"] Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136112 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136124 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136151 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d8f4153-d59a-4423-85ca-eb18263169f5" containerName="revoke-keys" Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136156 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d8f4153-d59a-4423-85ca-eb18263169f5" containerName="revoke-keys" Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136213 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136222 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d8f4153-d59a-4423-85ca-eb18263169f5" containerName="revoke-keys" Jul 28 18:30:00.136280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.136230 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="fc347066-3dd6-43bf-ae68-ee108ba86351" containerName="istio-proxy" Jul 28 18:30:00.139336 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.139315 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" Jul 28 18:30:00.143155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.143120 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:30:00.151176 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.151149 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8"] Jul 28 18:30:00.151642 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.151616 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" Jul 28 18:30:00.306968 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.306937 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8"] Jul 28 18:30:00.308740 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:30:00.308713 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddacfaa5a_65a1_489e_a545_e40eebaab5bb.slice/crio-6d63518cacdd88c23b0aa6e115a483bb41676535ea0bfc2277eb09005d1b1d0c WatchSource:0}: Error finding container 6d63518cacdd88c23b0aa6e115a483bb41676535ea0bfc2277eb09005d1b1d0c: Status 404 returned error can't find the container with id 6d63518cacdd88c23b0aa6e115a483bb41676535ea0bfc2277eb09005d1b1d0c Jul 28 18:30:00.682728 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.682692 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" event={"ID":"dacfaa5a-65a1-489e-a545-e40eebaab5bb","Type":"ContainerStarted","Data":"d435e36f7d8bfef54d0dc85b883cd48825e85743b96e8d74f729fea93086b6d5"} Jul 28 18:30:00.682906 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.682747 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" event={"ID":"dacfaa5a-65a1-489e-a545-e40eebaab5bb","Type":"ContainerStarted","Data":"6d63518cacdd88c23b0aa6e115a483bb41676535ea0bfc2277eb09005d1b1d0c"} Jul 28 18:30:00.698298 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:00.698217 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" podStartSLOduration=0.698201735 podStartE2EDuration="698.201735ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:30:00.697902656 +0000 UTC m=+1998.112559657" watchObservedRunningTime="2026-07-28 18:30:00.698201735 +0000 UTC m=+1998.112858737" Jul 28 18:30:01.689969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:01.689933 2575 generic.go:356] "Generic (PLEG): container finished" podID="dacfaa5a-65a1-489e-a545-e40eebaab5bb" containerID="d435e36f7d8bfef54d0dc85b883cd48825e85743b96e8d74f729fea93086b6d5" exitCode=0 Jul 28 18:30:01.690361 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:01.690013 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" event={"ID":"dacfaa5a-65a1-489e-a545-e40eebaab5bb","Type":"ContainerDied","Data":"d435e36f7d8bfef54d0dc85b883cd48825e85743b96e8d74f729fea93086b6d5"} Jul 28 18:30:02.830041 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:02.830007 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" Jul 28 18:30:03.702355 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:03.702319 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" Jul 28 18:30:03.702538 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:03.702353 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754390-47xd8" event={"ID":"dacfaa5a-65a1-489e-a545-e40eebaab5bb","Type":"ContainerDied","Data":"6d63518cacdd88c23b0aa6e115a483bb41676535ea0bfc2277eb09005d1b1d0c"} Jul 28 18:30:03.702538 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:03.702422 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d63518cacdd88c23b0aa6e115a483bb41676535ea0bfc2277eb09005d1b1d0c" Jul 28 18:30:06.940011 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:06.939976 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:13.896768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:13.896723 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:14.001222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:14.001174 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:14.127175 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:14.127123 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:30.133596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:30.133555 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:35.995071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:35.995035 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:36.109038 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:36.109008 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:36.217198 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:36.217166 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:45.347350 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:45.347316 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:46.787286 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:46.787251 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:47.630882 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:47.630842 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:54.698214 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:54.698180 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:54.802397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:54.802358 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:55.008081 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:55.007995 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:30:55.136898 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:30:55.136865 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:00.480276 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:00.480245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:00.573362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:00.573330 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:00.676739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:00.676700 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:02.713360 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:02.713327 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:08.979905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:08.979830 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:09.074434 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:09.074394 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:09.754886 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:09.754811 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:15.807898 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:15.807863 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:15.909180 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:15.909142 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:16.104998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.104917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:16.393291 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393249 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w"] Jul 28 18:31:16.393748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393720 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" Jul 28 18:31:16.393748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393740 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" Jul 28 18:31:16.393748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393749 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dacfaa5a-65a1-489e-a545-e40eebaab5bb" containerName="cleanup" Jul 28 18:31:16.393748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393755 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="dacfaa5a-65a1-489e-a545-e40eebaab5bb" containerName="cleanup" Jul 28 18:31:16.394052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393816 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="dacfaa5a-65a1-489e-a545-e40eebaab5bb" containerName="cleanup" Jul 28 18:31:16.394052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.393826 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="5936f75f-1443-41ed-b83b-0dd86394195a" containerName="istio-proxy" Jul 28 18:31:16.397081 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.397052 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.400818 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.400791 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 18:31:16.401004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.400791 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0540baca-openshift-default-dockercfg-pjnth\"" Jul 28 18:31:16.419166 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.419131 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w"] Jul 28 18:31:16.426738 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.426676 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:16.475326 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475283 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-credential-socket\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475508 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475335 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-podinfo\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475508 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475425 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvcf9\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-kube-api-access-cvcf9\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475508 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475463 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-socket\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475508 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475481 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-data\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475508 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475507 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-certs\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475622 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-envoy\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475659 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istiod-ca-cert\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.475716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.475693 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-token\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576680 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576637 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istiod-ca-cert\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576680 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576677 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-token\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-credential-socket\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576791 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-podinfo\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvcf9\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-kube-api-access-cvcf9\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576844 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-socket\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576860 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-data\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576899 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-certs\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.576986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.576945 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-envoy\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.577365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.577277 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-credential-socket\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.577424 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.577382 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-certs\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.577481 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.577422 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-data\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.577538 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.577496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-socket\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.577538 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.577520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istiod-ca-cert\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.579289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.579261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-envoy\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.579505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.579485 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-podinfo\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.587068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.587034 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-token\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.587241 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.587165 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvcf9\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-kube-api-access-cvcf9\") pod \"e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.604301 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.604269 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:16.709512 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.709423 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:16.859002 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.858975 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w"] Jul 28 18:31:16.860946 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:31:16.860909 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdee6522_292b_40d5_9ee1_0f765d0e0c93.slice/crio-83a2342c65d5732236c6f0a2090f80471d2474964269d58de0779ba606c8d5eb WatchSource:0}: Error finding container 83a2342c65d5732236c6f0a2090f80471d2474964269d58de0779ba606c8d5eb: Status 404 returned error can't find the container with id 83a2342c65d5732236c6f0a2090f80471d2474964269d58de0779ba606c8d5eb Jul 28 18:31:16.866157 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.866113 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:31:16.866255 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.866208 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:31:16.866299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:16.866257 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:31:17.129023 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:17.128983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" event={"ID":"bdee6522-292b-40d5-9ee1-0f765d0e0c93","Type":"ContainerStarted","Data":"5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf"} Jul 28 18:31:17.129190 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:17.129053 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" event={"ID":"bdee6522-292b-40d5-9ee1-0f765d0e0c93","Type":"ContainerStarted","Data":"83a2342c65d5732236c6f0a2090f80471d2474964269d58de0779ba606c8d5eb"} Jul 28 18:31:17.156102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:17.156041 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podStartSLOduration=1.156020013 podStartE2EDuration="1.156020013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:31:17.155275071 +0000 UTC m=+2074.569932086" watchObservedRunningTime="2026-07-28 18:31:17.156020013 +0000 UTC m=+2074.570677014" Jul 28 18:31:17.710479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:17.710435 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:17.715742 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:17.715712 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:18.136333 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:18.136301 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:18.137358 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:18.137336 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:19.393324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:19.393294 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:19.490129 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:19.490095 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:19.698155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:19.698066 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:19.802522 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:19.802483 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:20.443770 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:20.443722 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:20.543116 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:20.543082 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:33.137617 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.137581 2575 patch_prober.go:28] interesting pod/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 18:31:33.138056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.137643 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 18:31:33.222896 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.222865 2575 generic.go:356] "Generic (PLEG): container finished" podID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerID="5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf" exitCode=0 Jul 28 18:31:33.223066 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.222941 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" event={"ID":"bdee6522-292b-40d5-9ee1-0f765d0e0c93","Type":"ContainerDied","Data":"5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf"} Jul 28 18:31:33.223391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.223373 2575 scope.go:117] "RemoveContainer" containerID="5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf" Jul 28 18:31:33.224474 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.224445 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:31:33.224534 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.224520 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:31:33.224584 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:33.224572 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:31:34.230226 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:34.230186 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" event={"ID":"bdee6522-292b-40d5-9ee1-0f765d0e0c93","Type":"ContainerStarted","Data":"0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0"} Jul 28 18:31:34.709786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:34.709744 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:34.710015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:34.709992 2575 patch_prober.go:28] interesting pod/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 18:31:34.710109 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:34.710041 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 18:31:35.709951 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:35.709913 2575 patch_prober.go:28] interesting pod/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 18:31:35.710353 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:35.709973 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 18:31:36.710363 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:36.710316 2575 patch_prober.go:28] interesting pod/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 18:31:36.710802 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:36.710407 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 18:31:37.710543 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:37.710500 2575 patch_prober.go:28] interesting pod/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Jul 28 18:31:37.710969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:37.710556 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Jul 28 18:31:38.714350 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:38.714306 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:38.714809 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:38.714639 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:38.715487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:38.715464 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:31:44.431499 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:44.431454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:45.168409 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:45.168370 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:45.266092 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:45.266057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:45.685710 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:45.685644 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:51.703375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:51.703338 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:51.801679 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:51.801646 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:31:52.046390 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.046305 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j"] Jul 28 18:31:52.048820 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.048798 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:52.051419 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.051398 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:31:52.051489 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.051431 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:31:52.057986 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.057961 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j"] Jul 28 18:31:52.209837 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.209794 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e585a5-0539-4269-9bf0-2cba5fed0e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0540baca-pm87j\" (UID: \"b7e585a5-0539-4269-9bf0-2cba5fed0e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:52.311190 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.311105 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e585a5-0539-4269-9bf0-2cba5fed0e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0540baca-pm87j\" (UID: \"b7e585a5-0539-4269-9bf0-2cba5fed0e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:52.311827 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.311806 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e585a5-0539-4269-9bf0-2cba5fed0e14-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-0540baca-pm87j\" (UID: \"b7e585a5-0539-4269-9bf0-2cba5fed0e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:52.361106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.361067 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:52.511388 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:52.511361 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j"] Jul 28 18:31:52.513318 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:31:52.513283 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7e585a5_0539_4269_9bf0_2cba5fed0e14.slice/crio-7e91084e5b7fd7b42a7e58b779bfb726eaa4b5ef408d75246bfc8660a2ff70cc WatchSource:0}: Error finding container 7e91084e5b7fd7b42a7e58b779bfb726eaa4b5ef408d75246bfc8660a2ff70cc: Status 404 returned error can't find the container with id 7e91084e5b7fd7b42a7e58b779bfb726eaa4b5ef408d75246bfc8660a2ff70cc Jul 28 18:31:53.360295 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:53.360257 2575 generic.go:356] "Generic (PLEG): container finished" podID="b7e585a5-0539-4269-9bf0-2cba5fed0e14" containerID="4ccb729efb96cd7e946380537b9cb45f99accff6c7048a452299f5a9d5c2df20" exitCode=0 Jul 28 18:31:53.360676 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:53.360339 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" event={"ID":"b7e585a5-0539-4269-9bf0-2cba5fed0e14","Type":"ContainerDied","Data":"4ccb729efb96cd7e946380537b9cb45f99accff6c7048a452299f5a9d5c2df20"} Jul 28 18:31:53.360676 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:53.360392 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" event={"ID":"b7e585a5-0539-4269-9bf0-2cba5fed0e14","Type":"ContainerStarted","Data":"7e91084e5b7fd7b42a7e58b779bfb726eaa4b5ef408d75246bfc8660a2ff70cc"} Jul 28 18:31:54.498589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:54.498559 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:54.631670 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:54.631633 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e585a5-0539-4269-9bf0-2cba5fed0e14-maas-api-service-ca\") pod \"b7e585a5-0539-4269-9bf0-2cba5fed0e14\" (UID: \"b7e585a5-0539-4269-9bf0-2cba5fed0e14\") " Jul 28 18:31:54.632029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:54.632006 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7e585a5-0539-4269-9bf0-2cba5fed0e14-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b7e585a5-0539-4269-9bf0-2cba5fed0e14" (UID: "b7e585a5-0539-4269-9bf0-2cba5fed0e14"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:31:54.733303 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:54.733248 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7e585a5-0539-4269-9bf0-2cba5fed0e14-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:31:55.372610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:55.372523 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" Jul 28 18:31:55.372610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:55.372555 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j" event={"ID":"b7e585a5-0539-4269-9bf0-2cba5fed0e14","Type":"ContainerDied","Data":"7e91084e5b7fd7b42a7e58b779bfb726eaa4b5ef408d75246bfc8660a2ff70cc"} Jul 28 18:31:55.372610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:31:55.372604 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e91084e5b7fd7b42a7e58b779bfb726eaa4b5ef408d75246bfc8660a2ff70cc" Jul 28 18:32:02.802526 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:02.802490 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:02.967489 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:02.967456 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:03.066966 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:03.066868 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:03.120419 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:03.120383 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j"] Jul 28 18:32:03.123397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:03.123351 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0540baca-pm87j"] Jul 28 18:32:03.160502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:03.160472 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:03.509206 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:03.509170 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:05.058920 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:05.058877 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7e585a5-0539-4269-9bf0-2cba5fed0e14" path="/var/lib/kubelet/pods/b7e585a5-0539-4269-9bf0-2cba5fed0e14/volumes" Jul 28 18:32:09.423443 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:09.423407 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:09.523459 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:09.523426 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:09.672393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:09.672350 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w"] Jul 28 18:32:09.672593 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:09.672569 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" containerID="cri-o://0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0" gracePeriod=30 Jul 28 18:32:09.700381 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:09.700295 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:09.976203 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:09.976106 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:10.086888 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:10.086847 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:10.181864 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:10.181833 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:10.282046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:10.281957 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:10.523858 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:10.523814 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:10.638845 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:10.638814 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:14.927362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:14.927334 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:32:15.004414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004315 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-podinfo\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004358 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istiod-ca-cert\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004378 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-token\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004396 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-envoy\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004447 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-certs\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004491 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-credential-socket\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004536 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-socket\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004559 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-data\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.004788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cvcf9\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-kube-api-access-cvcf9\") pod \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\" (UID: \"bdee6522-292b-40d5-9ee1-0f765d0e0c93\") " Jul 28 18:32:15.005054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004778 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:32:15.005054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:32:15.005054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.004979 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:32:15.006768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.006738 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-kube-api-access-cvcf9" (OuterVolumeSpecName: "kube-api-access-cvcf9") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "kube-api-access-cvcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:32:15.007115 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.007087 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:32:15.007168 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.007089 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:32:15.007749 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.007725 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:32:15.007799 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.007775 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:32:15.008147 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.008126 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdee6522-292b-40d5-9ee1-0f765d0e0c93" (UID: "bdee6522-292b-40d5-9ee1-0f765d0e0c93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:32:15.105501 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105458 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105501 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105493 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105501 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105507 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvcf9\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-kube-api-access-cvcf9\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105516 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105525 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105534 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105541 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105550 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.105754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.105557 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdee6522-292b-40d5-9ee1-0f765d0e0c93-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:32:15.487046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.487012 2575 generic.go:356] "Generic (PLEG): container finished" podID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerID="0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0" exitCode=0 Jul 28 18:32:15.487230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.487058 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" Jul 28 18:32:15.487230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.487096 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" event={"ID":"bdee6522-292b-40d5-9ee1-0f765d0e0c93","Type":"ContainerDied","Data":"0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0"} Jul 28 18:32:15.487230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.487156 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w" event={"ID":"bdee6522-292b-40d5-9ee1-0f765d0e0c93","Type":"ContainerDied","Data":"83a2342c65d5732236c6f0a2090f80471d2474964269d58de0779ba606c8d5eb"} Jul 28 18:32:15.487230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.487180 2575 scope.go:117] "RemoveContainer" containerID="0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0" Jul 28 18:32:15.497030 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.497007 2575 scope.go:117] "RemoveContainer" containerID="5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf" Jul 28 18:32:15.504615 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.504579 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w"] Jul 28 18:32:15.507355 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.507331 2575 scope.go:117] "RemoveContainer" containerID="0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0" Jul 28 18:32:15.507666 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:32:15.507646 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0\": container with ID starting with 0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0 not found: ID does not exist" containerID="0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0" Jul 28 18:32:15.507748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.507676 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0"} err="failed to get container status \"0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0\": rpc error: code = NotFound desc = could not find container \"0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0\": container with ID starting with 0efaf3b7ba093c28efe550ae0a679d8306649cad187ba8e48e7b1426758004d0 not found: ID does not exist" Jul 28 18:32:15.507748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.507715 2575 scope.go:117] "RemoveContainer" containerID="5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf" Jul 28 18:32:15.508008 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:32:15.507993 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf\": container with ID starting with 5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf not found: ID does not exist" containerID="5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf" Jul 28 18:32:15.508053 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.508012 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf"} err="failed to get container status \"5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf\": rpc error: code = NotFound desc = could not find container \"5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf\": container with ID starting with 5d9412da98c756eabd6ec0dd6dadab75c6264086bd9aaf90927cf190e9a4d1cf not found: ID does not exist" Jul 28 18:32:15.508302 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.508282 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0540baca-openshift-default-6fdc8df848-jlv6w"] Jul 28 18:32:15.952329 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:15.952291 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:16.018111 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:16.018068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:16.103134 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:16.103099 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:16.201195 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:16.201159 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:16.320632 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:16.320541 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:17.059485 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:17.059443 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" path="/var/lib/kubelet/pods/bdee6522-292b-40d5-9ee1-0f765d0e0c93/volumes" Jul 28 18:32:19.144741 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:19.144703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:19.204605 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:19.204561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:22.130538 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:22.130486 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:22.259368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:22.259333 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:29.132370 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.132325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:29.432011 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.431910 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp"] Jul 28 18:32:29.432399 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432377 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" Jul 28 18:32:29.432487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432400 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" Jul 28 18:32:29.432487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432411 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" Jul 28 18:32:29.432487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432419 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" Jul 28 18:32:29.432487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432447 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7e585a5-0539-4269-9bf0-2cba5fed0e14" containerName="revoke-keys" Jul 28 18:32:29.432487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432456 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7e585a5-0539-4269-9bf0-2cba5fed0e14" containerName="revoke-keys" Jul 28 18:32:29.432806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432541 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" Jul 28 18:32:29.432806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432553 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdee6522-292b-40d5-9ee1-0f765d0e0c93" containerName="istio-proxy" Jul 28 18:32:29.432806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.432569 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7e585a5-0539-4269-9bf0-2cba5fed0e14" containerName="revoke-keys" Jul 28 18:32:29.434856 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.434833 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.437744 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.437717 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 18:32:29.437885 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.437726 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-06daccab-openshift-default-dockercfg-rp5q2\"" Jul 28 18:32:29.449884 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.449853 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp"] Jul 28 18:32:29.461413 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.461381 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:29.535795 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535746 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-podinfo\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.535795 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535798 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-socket\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535831 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-envoy\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535864 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-data\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535892 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5579z\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-kube-api-access-5579z\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535914 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-token\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535947 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-credential-socket\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.535978 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acabd454-e8dd-403e-a874-1ea1f70679d3-istiod-ca-cert\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.536200 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.536063 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-certs\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.636811 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.636772 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-credential-socket\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.636811 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.636822 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acabd454-e8dd-403e-a874-1ea1f70679d3-istiod-ca-cert\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.636854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-certs\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.636885 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-podinfo\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.636904 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-socket\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637003 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-envoy\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637052 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-data\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637095 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5579z\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-kube-api-access-5579z\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637130 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-token\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637273 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-credential-socket\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637295 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-certs\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637385 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-socket\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637593 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637562 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-data\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.637730 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.637678 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acabd454-e8dd-403e-a874-1ea1f70679d3-istiod-ca-cert\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.639380 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.639354 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-envoy\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.639591 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.639568 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-podinfo\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.648142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.648114 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-token\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.648294 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.648201 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5579z\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-kube-api-access-5579z\") pod \"e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.669361 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.669325 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:29.748599 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.748498 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:29.893187 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.893151 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp"] Jul 28 18:32:29.894209 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:32:29.894184 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacabd454_e8dd_403e_a874_1ea1f70679d3.slice/crio-670121bdcd011e120b64f921be6ed742694d6fee806c4b66a5793710309a2926 WatchSource:0}: Error finding container 670121bdcd011e120b64f921be6ed742694d6fee806c4b66a5793710309a2926: Status 404 returned error can't find the container with id 670121bdcd011e120b64f921be6ed742694d6fee806c4b66a5793710309a2926 Jul 28 18:32:29.896320 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.896290 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:32:29.896376 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.896351 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:32:29.896412 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:29.896378 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:32:30.576143 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:30.576106 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" event={"ID":"acabd454-e8dd-403e-a874-1ea1f70679d3","Type":"ContainerStarted","Data":"451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970"} Jul 28 18:32:30.576550 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:30.576159 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" event={"ID":"acabd454-e8dd-403e-a874-1ea1f70679d3","Type":"ContainerStarted","Data":"670121bdcd011e120b64f921be6ed742694d6fee806c4b66a5793710309a2926"} Jul 28 18:32:30.599437 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:30.599368 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podStartSLOduration=1.5993455600000002 podStartE2EDuration="1.59934556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:32:30.597570296 +0000 UTC m=+2148.012227297" watchObservedRunningTime="2026-07-28 18:32:30.59934556 +0000 UTC m=+2148.014002566" Jul 28 18:32:30.749447 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:30.749406 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:30.754604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:30.754561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:31.583230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:31.583190 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:31.584306 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:31.584284 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:32.516421 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:32.516376 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:32.616041 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:32.615983 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:32.764405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:32.764357 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:32.962473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:32.962435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:33.281810 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:33.281716 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:33.383399 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:33.383367 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:32:47.624885 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.624834 2575 patch_prober.go:28] interesting pod/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 18:32:47.626446 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.624903 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 18:32:47.689542 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.689507 2575 generic.go:356] "Generic (PLEG): container finished" podID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerID="451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970" exitCode=0 Jul 28 18:32:47.689768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.689580 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" event={"ID":"acabd454-e8dd-403e-a874-1ea1f70679d3","Type":"ContainerDied","Data":"451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970"} Jul 28 18:32:47.690075 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.690056 2575 scope.go:117] "RemoveContainer" containerID="451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970" Jul 28 18:32:47.691182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.691151 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:32:47.691240 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.691222 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:32:47.691277 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:47.691255 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:32:48.697203 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:48.697166 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" event={"ID":"acabd454-e8dd-403e-a874-1ea1f70679d3","Type":"ContainerStarted","Data":"d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535"} Jul 28 18:32:48.749217 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:48.749179 2575 patch_prober.go:28] interesting pod/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 18:32:48.749554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:48.749507 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 18:32:48.750037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:48.750015 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:49.750280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:49.749801 2575 patch_prober.go:28] interesting pod/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 18:32:49.750280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:49.749867 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 18:32:50.749318 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:50.749274 2575 patch_prober.go:28] interesting pod/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 18:32:50.749634 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:50.749602 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 18:32:51.749744 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:51.749666 2575 patch_prober.go:28] interesting pod/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Jul 28 18:32:51.750277 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:51.749771 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Jul 28 18:32:52.753012 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:52.752979 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:52.753398 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:52.753362 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:52.754136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:52.754119 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:32:57.581584 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:32:57.581547 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:06.319613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:06.319578 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:28.973153 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:28.973117 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:29.036976 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:29.036925 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:37.558918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.558879 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc"] Jul 28 18:33:37.561553 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.561534 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:37.564179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.564142 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:33:37.564179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.564142 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:33:37.570499 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.570471 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc"] Jul 28 18:33:37.572780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.572744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06daccab-jn2jc\" (UID: \"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:37.674140 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.674088 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06daccab-jn2jc\" (UID: \"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:37.674817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.674796 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-06daccab-jn2jc\" (UID: \"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:37.873547 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:37.873450 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:38.021586 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:38.021551 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc"] Jul 28 18:33:38.023539 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:33:38.023504 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff176e7_9dad_40e5_b28d_b3be6a30b6c7.slice/crio-4b8266586dd4019cfcfcea2049600ad97d16dc99e261aa4ff3ceb81c8c857c8c WatchSource:0}: Error finding container 4b8266586dd4019cfcfcea2049600ad97d16dc99e261aa4ff3ceb81c8c857c8c: Status 404 returned error can't find the container with id 4b8266586dd4019cfcfcea2049600ad97d16dc99e261aa4ff3ceb81c8c857c8c Jul 28 18:33:38.987998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:38.987894 2575 generic.go:356] "Generic (PLEG): container finished" podID="6ff176e7-9dad-40e5-b28d-b3be6a30b6c7" containerID="327a76abf459fb062b259f302a4bf253a809cd66e394a0833973528d71f2c15f" exitCode=0 Jul 28 18:33:38.987998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:38.987983 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" event={"ID":"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7","Type":"ContainerDied","Data":"327a76abf459fb062b259f302a4bf253a809cd66e394a0833973528d71f2c15f"} Jul 28 18:33:38.988422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:38.988047 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" event={"ID":"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7","Type":"ContainerStarted","Data":"4b8266586dd4019cfcfcea2049600ad97d16dc99e261aa4ff3ceb81c8c857c8c"} Jul 28 18:33:40.137146 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:40.137115 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:40.193594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:40.193550 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7-maas-api-service-ca\") pod \"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7\" (UID: \"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7\") " Jul 28 18:33:40.193972 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:40.193945 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6ff176e7-9dad-40e5-b28d-b3be6a30b6c7" (UID: "6ff176e7-9dad-40e5-b28d-b3be6a30b6c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:33:40.294368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:40.294267 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:33:41.002318 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:41.002287 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" Jul 28 18:33:41.002510 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:41.002312 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc" event={"ID":"6ff176e7-9dad-40e5-b28d-b3be6a30b6c7","Type":"ContainerDied","Data":"4b8266586dd4019cfcfcea2049600ad97d16dc99e261aa4ff3ceb81c8c857c8c"} Jul 28 18:33:41.002510 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:41.002364 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b8266586dd4019cfcfcea2049600ad97d16dc99e261aa4ff3ceb81c8c857c8c" Jul 28 18:33:43.367799 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:43.367756 2575 scope.go:117] "RemoveContainer" containerID="6fb6c7bcabed0cb8496348fbf316f90d282ed5e92f8e7afb9e4d486e5c651a8a" Jul 28 18:33:52.987352 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:52.987307 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:53.087940 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:53.087897 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:53.151625 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:53.151584 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc"] Jul 28 18:33:53.153279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:53.153252 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:33:53.154387 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:53.154364 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-06daccab-jn2jc"] Jul 28 18:33:55.065597 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:55.065557 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff176e7-9dad-40e5-b28d-b3be6a30b6c7" path="/var/lib/kubelet/pods/6ff176e7-9dad-40e5-b28d-b3be6a30b6c7/volumes" Jul 28 18:33:56.551353 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:33:56.551292 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:01.623816 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:01.623772 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:37.913135 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:37.913090 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:39.176836 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:39.176791 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:43.391038 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:43.391005 2575 scope.go:117] "RemoveContainer" containerID="9f734b2081da1748963026f711015e4112d28da5862b9de58eb91e0b893892fe" Jul 28 18:34:44.663746 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:44.663710 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:44.764905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:44.764871 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:44.931491 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:44.931398 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp"] Jul 28 18:34:44.931735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:44.931678 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" containerID="cri-o://d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535" gracePeriod=30 Jul 28 18:34:44.946220 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:44.946185 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:45.203233 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:45.203141 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:46.369973 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:46.369934 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:46.465197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:46.465164 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:47.340636 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:47.340605 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:47.427889 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:47.427848 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:47.530581 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:47.530552 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:49.018999 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:49.018904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:50.195046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.194979 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:34:50.318979 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.318942 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5579z\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-kube-api-access-5579z\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319164 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.318994 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-certs\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319164 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319024 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-podinfo\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319164 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319041 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-credential-socket\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319164 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319110 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-token\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319164 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319143 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acabd454-e8dd-403e-a874-1ea1f70679d3-istiod-ca-cert\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319458 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319179 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-data\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319458 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319204 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-socket\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319458 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319237 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-envoy\") pod \"acabd454-e8dd-403e-a874-1ea1f70679d3\" (UID: \"acabd454-e8dd-403e-a874-1ea1f70679d3\") " Jul 28 18:34:50.319458 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319308 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:50.319667 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319527 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.319667 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319440 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:50.319799 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319667 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acabd454-e8dd-403e-a874-1ea1f70679d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:34:50.319799 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.319772 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:50.320531 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.320484 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:50.321896 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.321868 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:34:50.322012 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.321896 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-kube-api-access-5579z" (OuterVolumeSpecName: "kube-api-access-5579z") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "kube-api-access-5579z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:34:50.322274 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.322244 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:34:50.322456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.322435 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "acabd454-e8dd-403e-a874-1ea1f70679d3" (UID: "acabd454-e8dd-403e-a874-1ea1f70679d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:34:50.401408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.401361 2575 generic.go:356] "Generic (PLEG): container finished" podID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerID="d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535" exitCode=0 Jul 28 18:34:50.401580 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.401404 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" Jul 28 18:34:50.401580 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.401428 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" event={"ID":"acabd454-e8dd-403e-a874-1ea1f70679d3","Type":"ContainerDied","Data":"d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535"} Jul 28 18:34:50.401580 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.401490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp" event={"ID":"acabd454-e8dd-403e-a874-1ea1f70679d3","Type":"ContainerDied","Data":"670121bdcd011e120b64f921be6ed742694d6fee806c4b66a5793710309a2926"} Jul 28 18:34:50.401580 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.401509 2575 scope.go:117] "RemoveContainer" containerID="d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535" Jul 28 18:34:50.417679 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.417657 2575 scope.go:117] "RemoveContainer" containerID="451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970" Jul 28 18:34:50.420131 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420102 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420134 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420151 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420166 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5579z\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-kube-api-access-5579z\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420180 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420194 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acabd454-e8dd-403e-a874-1ea1f70679d3-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420207 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acabd454-e8dd-403e-a874-1ea1f70679d3-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.420260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.420221 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acabd454-e8dd-403e-a874-1ea1f70679d3-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:34:50.428695 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.428657 2575 scope.go:117] "RemoveContainer" containerID="d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535" Jul 28 18:34:50.429103 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:34:50.429080 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535\": container with ID starting with d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535 not found: ID does not exist" containerID="d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535" Jul 28 18:34:50.429204 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.429112 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535"} err="failed to get container status \"d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535\": rpc error: code = NotFound desc = could not find container \"d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535\": container with ID starting with d45b77507c0959c476ddc6b8e3601407047a5a4944b3a5e496e2018ce107a535 not found: ID does not exist" Jul 28 18:34:50.429204 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.429132 2575 scope.go:117] "RemoveContainer" containerID="451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970" Jul 28 18:34:50.429452 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:34:50.429430 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970\": container with ID starting with 451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970 not found: ID does not exist" containerID="451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970" Jul 28 18:34:50.429537 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.429458 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970"} err="failed to get container status \"451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970\": rpc error: code = NotFound desc = could not find container \"451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970\": container with ID starting with 451d67afa69f6ccaa72effe744dfee6c2fd643e31609fe03fa73c745032aa970 not found: ID does not exist" Jul 28 18:34:50.434510 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.434477 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp"] Jul 28 18:34:50.435744 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:34:50.435586 2575 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacabd454_e8dd_403e_a874_1ea1f70679d3.slice\": RecentStats: unable to find data in memory cache]" Jul 28 18:34:50.439816 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:50.439781 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-06daccab-openshift-default-6856ffdc9b-4rdnp"] Jul 28 18:34:51.070376 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:51.070337 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" path="/var/lib/kubelet/pods/acabd454-e8dd-403e-a874-1ea1f70679d3/volumes" Jul 28 18:34:55.019230 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:55.019189 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:55.120554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:55.120517 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:34:55.715988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:34:55.715953 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:01.877940 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:01.877904 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:01.984768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:01.984737 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:02.078940 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:02.078903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:12.713633 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:12.713593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:13.079179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:13.079087 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:34.042698 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:34.042659 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:40.621779 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:40.621667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:40.721834 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:40.721794 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:40.988279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:40.988240 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:41.409996 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.409957 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn"] Jul 28 18:35:41.410340 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410327 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" Jul 28 18:35:41.410393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410342 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" Jul 28 18:35:41.410393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410356 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" Jul 28 18:35:41.410393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410362 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" Jul 28 18:35:41.410393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410376 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff176e7-9dad-40e5-b28d-b3be6a30b6c7" containerName="revoke-keys" Jul 28 18:35:41.410393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410382 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff176e7-9dad-40e5-b28d-b3be6a30b6c7" containerName="revoke-keys" Jul 28 18:35:41.410570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410442 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" Jul 28 18:35:41.410570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.410452 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff176e7-9dad-40e5-b28d-b3be6a30b6c7" containerName="revoke-keys" Jul 28 18:35:41.412999 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.412977 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.416099 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.416077 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 28 18:35:41.416830 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.416807 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b228ad-openshift-default-dockercfg-f9jwj\"" Jul 28 18:35:41.430395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.430356 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn"] Jul 28 18:35:41.435700 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.435633 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:41.478296 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478252 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc36c705-4974-4398-b15a-c6afafa60f05-istio-podinfo\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-credential-socket\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478363 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-istio-token\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478394 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-data\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478417 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-envoy\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478440 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7k2d\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-kube-api-access-p7k2d\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478652 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478479 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-socket\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478652 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478506 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc36c705-4974-4398-b15a-c6afafa60f05-istiod-ca-cert\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.478652 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.478528 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-certs\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579617 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579570 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-socket\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579617 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579623 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc36c705-4974-4398-b15a-c6afafa60f05-istiod-ca-cert\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579656 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-certs\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579711 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc36c705-4974-4398-b15a-c6afafa60f05-istio-podinfo\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579752 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-credential-socket\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-istio-token\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579846 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-data\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.579903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579875 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-envoy\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.580222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.579915 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7k2d\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-kube-api-access-p7k2d\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.580222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.580111 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-socket\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.580373 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.580346 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-credential-socket\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.580431 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.580366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-certs\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.580514 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.580496 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc36c705-4974-4398-b15a-c6afafa60f05-istiod-ca-cert\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.580561 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.580513 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-data\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.582342 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.582318 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-envoy\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.582488 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.582468 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc36c705-4974-4398-b15a-c6afafa60f05-istio-podinfo\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.589748 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.589717 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-istio-token\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.589959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.589935 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7k2d\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-kube-api-access-p7k2d\") pod \"e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.710660 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.710562 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:41.724397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.724357 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:41.870251 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.870220 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn"] Jul 28 18:35:41.871430 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:35:41.871394 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc36c705_4974_4398_b15a_c6afafa60f05.slice/crio-64e535cc820a2d9ede341ea13ac5eb972a64ee903f8aaa57d48d830dfccc538c WatchSource:0}: Error finding container 64e535cc820a2d9ede341ea13ac5eb972a64ee903f8aaa57d48d830dfccc538c: Status 404 returned error can't find the container with id 64e535cc820a2d9ede341ea13ac5eb972a64ee903f8aaa57d48d830dfccc538c Jul 28 18:35:41.873580 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.873540 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:35:41.873652 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.873631 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:35:41.873724 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:41.873677 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:35:42.709791 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:42.709749 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" event={"ID":"bc36c705-4974-4398-b15a-c6afafa60f05","Type":"ContainerStarted","Data":"7c797a24e4e229f0ae05a4a27e04aa020badd86ad1abe241589d3ce0eb01fecb"} Jul 28 18:35:42.710016 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:42.709819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" event={"ID":"bc36c705-4974-4398-b15a-c6afafa60f05","Type":"ContainerStarted","Data":"64e535cc820a2d9ede341ea13ac5eb972a64ee903f8aaa57d48d830dfccc538c"} Jul 28 18:35:42.724982 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:42.724939 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:42.730462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:42.730428 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:42.735398 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:42.735340 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" podStartSLOduration=1.735322625 podStartE2EDuration="1.735322625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:35:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:35:42.733143859 +0000 UTC m=+2340.147800860" watchObservedRunningTime="2026-07-28 18:35:42.735322625 +0000 UTC m=+2340.149979629" Jul 28 18:35:43.413667 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:43.413616 2575 scope.go:117] "RemoveContainer" containerID="bcb4c1e68356e7720b6b695de671fd8c3cfad5200d7d5e4ed160efb33479b4c4" Jul 28 18:35:43.423193 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:43.423163 2575 scope.go:117] "RemoveContainer" containerID="9967fda620b3d5aaeea106678c528391dae65797b9b88272d7f9aac613d7ae36" Jul 28 18:35:43.718627 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:43.718528 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:43.719672 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:43.719637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:35:44.702479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:44.702438 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:44.807089 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:44.807050 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:44.903379 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:44.903337 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:45.030929 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:45.030840 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:45.473344 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:45.473311 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:45.573723 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:45.573666 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:35:59.007456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.006889 2575 patch_prober.go:28] interesting pod/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:38220->10.132.0.62:15021: read: connection reset by peer" start-of-body= Jul 28 18:35:59.007456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.006974 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": read tcp 10.132.0.2:38220->10.132.0.62:15021: read: connection reset by peer" Jul 28 18:35:59.816027 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.815989 2575 generic.go:356] "Generic (PLEG): container finished" podID="bc36c705-4974-4398-b15a-c6afafa60f05" containerID="7c797a24e4e229f0ae05a4a27e04aa020badd86ad1abe241589d3ce0eb01fecb" exitCode=0 Jul 28 18:35:59.816205 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.816059 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" event={"ID":"bc36c705-4974-4398-b15a-c6afafa60f05","Type":"ContainerDied","Data":"7c797a24e4e229f0ae05a4a27e04aa020badd86ad1abe241589d3ce0eb01fecb"} Jul 28 18:35:59.816457 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.816444 2575 scope.go:117] "RemoveContainer" containerID="7c797a24e4e229f0ae05a4a27e04aa020badd86ad1abe241589d3ce0eb01fecb" Jul 28 18:35:59.817487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.817461 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:35:59.817536 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.817527 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:35:59.817575 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:35:59.817554 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:00.823536 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:00.823499 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" event={"ID":"bc36c705-4974-4398-b15a-c6afafa60f05","Type":"ContainerStarted","Data":"c50fb45c88880d5dca183f889711246bee5a6f47fdf2c01be7cdc48e0c28cba2"} Jul 28 18:36:01.725447 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:01.725408 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:36:01.725658 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:01.725635 2575 patch_prober.go:28] interesting pod/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 28 18:36:01.725734 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:01.725706 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 28 18:36:02.724863 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:02.724822 2575 patch_prober.go:28] interesting pod/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 28 18:36:02.725273 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:02.724884 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 28 18:36:03.725321 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:03.725278 2575 patch_prober.go:28] interesting pod/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 28 18:36:03.725759 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:03.725336 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 28 18:36:04.729344 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:04.729310 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:36:04.729902 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:04.729596 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:36:04.730214 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:04.730194 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:36:14.864089 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:14.864053 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:15.260244 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.260210 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:15.501208 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.501167 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4"] Jul 28 18:36:15.501622 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.501608 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="acabd454-e8dd-403e-a874-1ea1f70679d3" containerName="istio-proxy" Jul 28 18:36:15.505006 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.504986 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.507618 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.507589 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-210335-openshift-default-dockercfg-ww546\"" Jul 28 18:36:15.517807 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.517769 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4"] Jul 28 18:36:15.525009 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.524980 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:15.575056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575019 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-credential-socket\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575060 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-data\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575087 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfj55\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-kube-api-access-wfj55\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575109 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-podinfo\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575168 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-token\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575237 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-certs\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575286 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-socket\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575330 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-envoy\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.575590 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.575373 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istiod-ca-cert\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676283 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676237 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfj55\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-kube-api-access-wfj55\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676283 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676287 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-podinfo\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676305 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-token\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676333 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-certs\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676358 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-socket\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-envoy\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676427 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istiod-ca-cert\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-credential-socket\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676525 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-data\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676971 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676843 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-certs\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.676971 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676889 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-socket\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.677052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.676989 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-data\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.677052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.677036 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-credential-socket\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.677273 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.677250 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istiod-ca-cert\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.678916 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.678887 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-envoy\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.679298 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.679276 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-podinfo\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.684914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.684888 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-token\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.685055 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.684960 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfj55\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-kube-api-access-wfj55\") pod \"e2e-inf-b-210335-openshift-default-7669d7c896-zzft4\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.816883 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.816783 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:15.922097 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.922057 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:15.967516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.967484 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4"] Jul 28 18:36:15.968723 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:36:15.968667 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6886918f_844a_4d85_b4b6_5c389c5dd1f7.slice/crio-271bfddbd957590da3cf9595bbfaecac372aaa9434acc991d5cc00f80a46f0f5 WatchSource:0}: Error finding container 271bfddbd957590da3cf9595bbfaecac372aaa9434acc991d5cc00f80a46f0f5: Status 404 returned error can't find the container with id 271bfddbd957590da3cf9595bbfaecac372aaa9434acc991d5cc00f80a46f0f5 Jul 28 18:36:15.974247 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.974205 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:15.974385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.974283 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:15.974385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:15.974310 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:16.926253 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:16.926208 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" event={"ID":"6886918f-844a-4d85-b4b6-5c389c5dd1f7","Type":"ContainerStarted","Data":"43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976"} Jul 28 18:36:16.926654 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:16.926311 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" event={"ID":"6886918f-844a-4d85-b4b6-5c389c5dd1f7","Type":"ContainerStarted","Data":"271bfddbd957590da3cf9595bbfaecac372aaa9434acc991d5cc00f80a46f0f5"} Jul 28 18:36:16.949142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:16.949081 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podStartSLOduration=1.949061771 podStartE2EDuration="1.949061771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:36:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:36:16.946507209 +0000 UTC m=+2374.361164210" watchObservedRunningTime="2026-07-28 18:36:16.949061771 +0000 UTC m=+2374.363718847" Jul 28 18:36:17.817244 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:17.817202 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:17.822161 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:17.822132 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:17.932576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:17.932543 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:17.933602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:17.933581 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:18.891429 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:18.891321 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:19.005705 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:19.005660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:19.097249 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:19.097206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:19.402483 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:19.402449 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:19.591903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:19.591864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:19.697788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:19.697701 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:33.973918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:33.973864 2575 patch_prober.go:28] interesting pod/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 18:36:33.975144 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:33.973947 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 18:36:35.033442 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:35.033404 2575 generic.go:356] "Generic (PLEG): container finished" podID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerID="43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976" exitCode=0 Jul 28 18:36:35.033912 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:35.033480 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" event={"ID":"6886918f-844a-4d85-b4b6-5c389c5dd1f7","Type":"ContainerDied","Data":"43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976"} Jul 28 18:36:35.033912 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:35.033909 2575 scope.go:117] "RemoveContainer" containerID="43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976" Jul 28 18:36:35.035020 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:35.034988 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:35.035094 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:35.035068 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:35.035136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:35.035103 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:36:36.047091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:36.047049 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" event={"ID":"6886918f-844a-4d85-b4b6-5c389c5dd1f7","Type":"ContainerStarted","Data":"2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76"} Jul 28 18:36:36.817680 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:36.817637 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:36.818024 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:36.817998 2575 patch_prober.go:28] interesting pod/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 18:36:36.818134 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:36.818055 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 18:36:37.817486 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:37.817444 2575 patch_prober.go:28] interesting pod/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 18:36:37.817939 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:37.817503 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 18:36:38.818091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:38.818054 2575 patch_prober.go:28] interesting pod/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 18:36:38.818503 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:38.818113 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 18:36:39.817988 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:39.817950 2575 patch_prober.go:28] interesting pod/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Jul 28 18:36:39.818188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:39.818009 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Jul 28 18:36:40.821809 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:40.821773 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:40.822206 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:40.822155 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:40.822954 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:40.822930 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:36:43.860185 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:43.860137 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:36:48.812627 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:36:48.812588 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:37:04.278585 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:37:04.278554 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:37:23.810894 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:37:23.810857 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:04.029836 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:04.029798 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:04.964355 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:04.964320 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:05.069205 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.069172 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv"] Jul 28 18:38:05.072731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.072713 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:05.076075 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.076046 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:38:05.076241 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.076092 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:38:05.082221 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.082186 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv"] Jul 28 18:38:05.260092 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.259996 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a844d64c-2541-4403-8095-f176787c67da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv\" (UID: \"a844d64c-2541-4403-8095-f176787c67da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:05.361337 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.361288 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a844d64c-2541-4403-8095-f176787c67da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv\" (UID: \"a844d64c-2541-4403-8095-f176787c67da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:05.361942 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.361924 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a844d64c-2541-4403-8095-f176787c67da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv\" (UID: \"a844d64c-2541-4403-8095-f176787c67da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:05.384856 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.384820 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:05.525346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.525309 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv"] Jul 28 18:38:05.526842 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:38:05.526814 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda844d64c_2541_4403_8095_f176787c67da.slice/crio-638ce57c0084e9cef6574091f3300180ae5287d85da63c8bfba0f8c8c193a2b6 WatchSource:0}: Error finding container 638ce57c0084e9cef6574091f3300180ae5287d85da63c8bfba0f8c8c193a2b6: Status 404 returned error can't find the container with id 638ce57c0084e9cef6574091f3300180ae5287d85da63c8bfba0f8c8c193a2b6 Jul 28 18:38:05.572554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:05.572520 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" event={"ID":"a844d64c-2541-4403-8095-f176787c67da","Type":"ContainerStarted","Data":"638ce57c0084e9cef6574091f3300180ae5287d85da63c8bfba0f8c8c193a2b6"} Jul 28 18:38:06.579268 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:06.579171 2575 generic.go:356] "Generic (PLEG): container finished" podID="a844d64c-2541-4403-8095-f176787c67da" containerID="289a6c7b4fce3e8a8f9aeee4d30b95e1d32ee5c08df97787b17b9d054eba0117" exitCode=0 Jul 28 18:38:06.579632 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:06.579267 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" event={"ID":"a844d64c-2541-4403-8095-f176787c67da","Type":"ContainerDied","Data":"289a6c7b4fce3e8a8f9aeee4d30b95e1d32ee5c08df97787b17b9d054eba0117"} Jul 28 18:38:07.726235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:07.726197 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:07.780942 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:07.780898 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a844d64c-2541-4403-8095-f176787c67da-maas-api-service-ca\") pod \"a844d64c-2541-4403-8095-f176787c67da\" (UID: \"a844d64c-2541-4403-8095-f176787c67da\") " Jul 28 18:38:07.781290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:07.781265 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a844d64c-2541-4403-8095-f176787c67da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a844d64c-2541-4403-8095-f176787c67da" (UID: "a844d64c-2541-4403-8095-f176787c67da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:38:07.882264 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:07.882229 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a844d64c-2541-4403-8095-f176787c67da-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:08.592708 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:08.592667 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" Jul 28 18:38:08.592887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:08.592711 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv" event={"ID":"a844d64c-2541-4403-8095-f176787c67da","Type":"ContainerDied","Data":"638ce57c0084e9cef6574091f3300180ae5287d85da63c8bfba0f8c8c193a2b6"} Jul 28 18:38:08.592887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:08.592761 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="638ce57c0084e9cef6574091f3300180ae5287d85da63c8bfba0f8c8c193a2b6" Jul 28 18:38:15.606218 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:15.606179 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:15.675707 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:15.675654 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:15.746668 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:15.746635 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:15.791874 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:15.791834 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv"] Jul 28 18:38:15.794926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:15.794896 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b228ad-dh6dv"] Jul 28 18:38:15.809867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:15.809832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:16.218850 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:16.218809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:17.059210 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:17.059163 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a844d64c-2541-4403-8095-f176787c67da" path="/var/lib/kubelet/pods/a844d64c-2541-4403-8095-f176787c67da/volumes" Jul 28 18:38:22.276602 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.276561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:22.376982 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.376949 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:22.530628 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.530544 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn"] Jul 28 18:38:22.530837 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.530815 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" containerID="cri-o://c50fb45c88880d5dca183f889711246bee5a6f47fdf2c01be7cdc48e0c28cba2" gracePeriod=30 Jul 28 18:38:22.552098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.552059 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:22.800803 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.800712 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:22.911299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.911241 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w"] Jul 28 18:38:22.911745 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.911726 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a844d64c-2541-4403-8095-f176787c67da" containerName="revoke-keys" Jul 28 18:38:22.911858 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.911747 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="a844d64c-2541-4403-8095-f176787c67da" containerName="revoke-keys" Jul 28 18:38:22.911916 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.911879 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="a844d64c-2541-4403-8095-f176787c67da" containerName="revoke-keys" Jul 28 18:38:22.914994 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.914971 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:22.917654 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.917629 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:38:22.917654 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.917645 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:38:22.923510 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:22.923485 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w"] Jul 28 18:38:23.018370 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.018311 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/053f4bad-c989-48ac-8efe-56448afabd5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-210335-dl87w\" (UID: \"053f4bad-c989-48ac-8efe-56448afabd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:23.119959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.119854 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/053f4bad-c989-48ac-8efe-56448afabd5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-210335-dl87w\" (UID: \"053f4bad-c989-48ac-8efe-56448afabd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:23.120484 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.120462 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/053f4bad-c989-48ac-8efe-56448afabd5c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-210335-dl87w\" (UID: \"053f4bad-c989-48ac-8efe-56448afabd5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:23.227631 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.227591 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:23.371098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.371071 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w"] Jul 28 18:38:23.373785 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:38:23.373755 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod053f4bad_c989_48ac_8efe_56448afabd5c.slice/crio-57f48ddfa416f55d6f433a839358a4911303aa78a91d8bddb70201b11dbf4cbe WatchSource:0}: Error finding container 57f48ddfa416f55d6f433a839358a4911303aa78a91d8bddb70201b11dbf4cbe: Status 404 returned error can't find the container with id 57f48ddfa416f55d6f433a839358a4911303aa78a91d8bddb70201b11dbf4cbe Jul 28 18:38:23.683844 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.683800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" event={"ID":"053f4bad-c989-48ac-8efe-56448afabd5c","Type":"ContainerStarted","Data":"4e657f05feb0572adf5400ab655311be7d656b3c8f47892ea8f88d42067d76df"} Jul 28 18:38:23.684017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.683862 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" event={"ID":"053f4bad-c989-48ac-8efe-56448afabd5c","Type":"ContainerStarted","Data":"57f48ddfa416f55d6f433a839358a4911303aa78a91d8bddb70201b11dbf4cbe"} Jul 28 18:38:23.701611 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:23.701554 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" podStartSLOduration=1.7015368149999999 podStartE2EDuration="1.701536815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:38:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:38:23.700370422 +0000 UTC m=+2501.115027420" watchObservedRunningTime="2026-07-28 18:38:23.701536815 +0000 UTC m=+2501.116193817" Jul 28 18:38:24.690638 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:24.690605 2575 generic.go:356] "Generic (PLEG): container finished" podID="053f4bad-c989-48ac-8efe-56448afabd5c" containerID="4e657f05feb0572adf5400ab655311be7d656b3c8f47892ea8f88d42067d76df" exitCode=0 Jul 28 18:38:24.691052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:24.690664 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" event={"ID":"053f4bad-c989-48ac-8efe-56448afabd5c","Type":"ContainerDied","Data":"4e657f05feb0572adf5400ab655311be7d656b3c8f47892ea8f88d42067d76df"} Jul 28 18:38:25.829740 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:25.829716 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:25.944414 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:25.944371 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/053f4bad-c989-48ac-8efe-56448afabd5c-maas-api-service-ca\") pod \"053f4bad-c989-48ac-8efe-56448afabd5c\" (UID: \"053f4bad-c989-48ac-8efe-56448afabd5c\") " Jul 28 18:38:25.944834 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:25.944805 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/053f4bad-c989-48ac-8efe-56448afabd5c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "053f4bad-c989-48ac-8efe-56448afabd5c" (UID: "053f4bad-c989-48ac-8efe-56448afabd5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:38:26.045809 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:26.045717 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/053f4bad-c989-48ac-8efe-56448afabd5c-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:26.702270 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:26.702242 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" Jul 28 18:38:26.702270 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:26.702252 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w" event={"ID":"053f4bad-c989-48ac-8efe-56448afabd5c","Type":"ContainerDied","Data":"57f48ddfa416f55d6f433a839358a4911303aa78a91d8bddb70201b11dbf4cbe"} Jul 28 18:38:26.702486 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:26.702294 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57f48ddfa416f55d6f433a839358a4911303aa78a91d8bddb70201b11dbf4cbe" Jul 28 18:38:27.711487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.711451 2575 generic.go:356] "Generic (PLEG): container finished" podID="bc36c705-4974-4398-b15a-c6afafa60f05" containerID="c50fb45c88880d5dca183f889711246bee5a6f47fdf2c01be7cdc48e0c28cba2" exitCode=0 Jul 28 18:38:27.711959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.711529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" event={"ID":"bc36c705-4974-4398-b15a-c6afafa60f05","Type":"ContainerDied","Data":"c50fb45c88880d5dca183f889711246bee5a6f47fdf2c01be7cdc48e0c28cba2"} Jul 28 18:38:27.711959 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.711613 2575 scope.go:117] "RemoveContainer" containerID="7c797a24e4e229f0ae05a4a27e04aa020badd86ad1abe241589d3ce0eb01fecb" Jul 28 18:38:27.790082 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.790054 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:38:27.862486 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862390 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc36c705-4974-4398-b15a-c6afafa60f05-istiod-ca-cert\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862486 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862477 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-data\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862674 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862505 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7k2d\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-kube-api-access-p7k2d\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862674 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862534 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-credential-socket\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862795 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862720 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-istio-token\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862833 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-socket\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862874 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-certs\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862995 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862893 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:27.862995 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862904 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc36c705-4974-4398-b15a-c6afafa60f05-istio-podinfo\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.862995 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862924 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc36c705-4974-4398-b15a-c6afafa60f05-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:38:27.862995 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.862960 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-envoy\") pod \"bc36c705-4974-4398-b15a-c6afafa60f05\" (UID: \"bc36c705-4974-4398-b15a-c6afafa60f05\") " Jul 28 18:38:27.863215 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863163 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:27.863215 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863184 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:27.863329 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863300 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.863329 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863313 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc36c705-4974-4398-b15a-c6afafa60f05-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.863329 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863322 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.863479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863331 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.863640 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.863614 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-data" (OuterVolumeSpecName: "istio-data") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:27.864905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.864877 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-kube-api-access-p7k2d" (OuterVolumeSpecName: "kube-api-access-p7k2d") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "kube-api-access-p7k2d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:38:27.865060 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.865040 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:27.865279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.865249 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-istio-token" (OuterVolumeSpecName: "istio-token") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:38:27.865279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.865254 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc36c705-4974-4398-b15a-c6afafa60f05-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bc36c705-4974-4398-b15a-c6afafa60f05" (UID: "bc36c705-4974-4398-b15a-c6afafa60f05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:38:27.964077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.964033 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc36c705-4974-4398-b15a-c6afafa60f05-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.964077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.964068 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.964077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.964078 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc36c705-4974-4398-b15a-c6afafa60f05-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.964077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.964086 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7k2d\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-kube-api-access-p7k2d\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:27.964347 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:27.964095 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc36c705-4974-4398-b15a-c6afafa60f05-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:28.719064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:28.719034 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" Jul 28 18:38:28.719527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:28.719080 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn" event={"ID":"bc36c705-4974-4398-b15a-c6afafa60f05","Type":"ContainerDied","Data":"64e535cc820a2d9ede341ea13ac5eb972a64ee903f8aaa57d48d830dfccc538c"} Jul 28 18:38:28.719527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:28.719139 2575 scope.go:117] "RemoveContainer" containerID="c50fb45c88880d5dca183f889711246bee5a6f47fdf2c01be7cdc48e0c28cba2" Jul 28 18:38:28.746631 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:28.746596 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn"] Jul 28 18:38:28.751017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:28.750977 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b228ad-openshift-default-599c5cbbbd-q49zn"] Jul 28 18:38:29.060095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:29.060006 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" path="/var/lib/kubelet/pods/bc36c705-4974-4398-b15a-c6afafa60f05/volumes" Jul 28 18:38:33.306903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.306871 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:33.406202 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.406147 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:33.472887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.472846 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w"] Jul 28 18:38:33.475271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.475243 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:33.475933 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.475912 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-210335-dl87w"] Jul 28 18:38:33.533089 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.533039 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:33.855647 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:33.855612 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:35.059868 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:35.059771 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="053f4bad-c989-48ac-8efe-56448afabd5c" path="/var/lib/kubelet/pods/053f4bad-c989-48ac-8efe-56448afabd5c/volumes" Jul 28 18:38:36.493631 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:36.493592 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:39.730966 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:39.730917 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:39.805871 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:39.805829 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:39.991887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:39.991797 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4"] Jul 28 18:38:39.992038 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:39.992009 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" containerID="cri-o://2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76" gracePeriod=30 Jul 28 18:38:40.015159 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.015122 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:40.291777 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.291655 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:40.614461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.614374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:40.824629 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.824583 2575 patch_prober.go:28] interesting pod/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:38:40.825031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.824657 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:38:40.915226 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915192 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg"] Jul 28 18:38:40.915646 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915628 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" Jul 28 18:38:40.915760 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915650 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" Jul 28 18:38:40.915760 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915698 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="053f4bad-c989-48ac-8efe-56448afabd5c" containerName="revoke-keys" Jul 28 18:38:40.915760 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915708 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="053f4bad-c989-48ac-8efe-56448afabd5c" containerName="revoke-keys" Jul 28 18:38:40.915760 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915720 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" Jul 28 18:38:40.915760 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915729 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" Jul 28 18:38:40.916027 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915843 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" Jul 28 18:38:40.916027 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915860 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="053f4bad-c989-48ac-8efe-56448afabd5c" containerName="revoke-keys" Jul 28 18:38:40.916027 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.915873 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc36c705-4974-4398-b15a-c6afafa60f05" containerName="istio-proxy" Jul 28 18:38:40.920596 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.920574 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:40.922968 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.922949 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a7357c-openshift-default-dockercfg-89w2b\"" Jul 28 18:38:40.933658 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.933631 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg"] Jul 28 18:38:40.952312 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:40.952278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:41.078997 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.078962 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88lgv\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-kube-api-access-88lgv\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079185 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079015 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-envoy\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079185 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079058 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c55657-5059-47fb-9f0a-7c453b953797-istiod-ca-cert\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079185 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079091 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-credential-socket\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079185 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079130 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-certs\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079185 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079151 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c55657-5059-47fb-9f0a-7c453b953797-istio-podinfo\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079424 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079204 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-socket\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079424 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079230 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-istio-token\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.079424 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.079301 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-data\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.117904 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.117860 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:41.180142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180038 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-socket\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180075 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-istio-token\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-data\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180170 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88lgv\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-kube-api-access-88lgv\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180205 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-envoy\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180232 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c55657-5059-47fb-9f0a-7c453b953797-istiod-ca-cert\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180262 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-credential-socket\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180300 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-certs\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180435 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180321 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c55657-5059-47fb-9f0a-7c453b953797-istio-podinfo\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180725 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180536 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-socket\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180725 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180573 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-data\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180725 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180649 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-credential-socket\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.180919 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.180897 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-certs\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.181028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.181010 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c55657-5059-47fb-9f0a-7c453b953797-istiod-ca-cert\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.182693 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.182658 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-envoy\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.182848 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.182832 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c55657-5059-47fb-9f0a-7c453b953797-istio-podinfo\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.189630 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.189603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-istio-token\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.189785 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.189767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88lgv\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-kube-api-access-88lgv\") pod \"e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.234009 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.233969 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:41.407254 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.407218 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg"] Jul 28 18:38:41.407576 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:38:41.407547 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74c55657_5059_47fb_9f0a_7c453b953797.slice/crio-703ece3805be534608a05acbf6c785765d965f5b81a638bb525ede0bd842e21a WatchSource:0}: Error finding container 703ece3805be534608a05acbf6c785765d965f5b81a638bb525ede0bd842e21a: Status 404 returned error can't find the container with id 703ece3805be534608a05acbf6c785765d965f5b81a638bb525ede0bd842e21a Jul 28 18:38:41.409866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.409828 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:38:41.409946 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.409919 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:38:41.409984 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.409954 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:38:41.798189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.798100 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" event={"ID":"74c55657-5059-47fb-9f0a-7c453b953797","Type":"ContainerStarted","Data":"e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb"} Jul 28 18:38:41.798189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.798158 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" event={"ID":"74c55657-5059-47fb-9f0a-7c453b953797","Type":"ContainerStarted","Data":"703ece3805be534608a05acbf6c785765d965f5b81a638bb525ede0bd842e21a"} Jul 28 18:38:41.821765 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:41.821710 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podStartSLOduration=1.821670948 podStartE2EDuration="1.821670948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:38:41.82034697 +0000 UTC m=+2519.235003971" watchObservedRunningTime="2026-07-28 18:38:41.821670948 +0000 UTC m=+2519.236327949" Jul 28 18:38:42.235048 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:42.235007 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:42.239957 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:42.239926 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:42.803925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:42.803893 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:42.805084 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:42.805056 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:38:43.472260 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:43.472225 2575 scope.go:117] "RemoveContainer" containerID="4ccb729efb96cd7e946380537b9cb45f99accff6c7048a452299f5a9d5c2df20" Jul 28 18:38:43.827549 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:43.827454 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:43.926470 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:43.926435 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:44.026024 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:44.025990 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:44.468517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:44.468481 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:44.567933 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:44.567898 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:44.654543 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:44.654505 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:45.241153 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.241123 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:38:45.424258 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424222 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-credential-socket\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424258 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424258 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-data\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424299 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-podinfo\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424322 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-socket\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424347 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istiod-ca-cert\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424384 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfj55\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-kube-api-access-wfj55\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424404 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-certs\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424436 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-token\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424468 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-envoy\") pod \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\" (UID: \"6886918f-844a-4d85-b4b6-5c389c5dd1f7\") " Jul 28 18:38:45.424883 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424547 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:45.424883 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424635 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:45.424883 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424761 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:45.424883 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424878 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.425090 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424899 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.425090 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.424918 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.425200 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.425121 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:38:45.425200 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.425155 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-data" (OuterVolumeSpecName: "istio-data") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:45.426634 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.426601 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-token" (OuterVolumeSpecName: "istio-token") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:38:45.426634 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.426626 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:38:45.426809 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.426633 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-kube-api-access-wfj55" (OuterVolumeSpecName: "kube-api-access-wfj55") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "kube-api-access-wfj55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:38:45.426809 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.426749 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6886918f-844a-4d85-b4b6-5c389c5dd1f7" (UID: "6886918f-844a-4d85-b4b6-5c389c5dd1f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:38:45.525754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.525714 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.525754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.525748 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.525754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.525757 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.525754 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.525766 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfj55\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-kube-api-access-wfj55\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.526028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.525775 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.526028 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.525783 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6886918f-844a-4d85-b4b6-5c389c5dd1f7-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:38:45.825293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.825202 2575 generic.go:356] "Generic (PLEG): container finished" podID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerID="2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76" exitCode=0 Jul 28 18:38:45.825293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.825254 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" Jul 28 18:38:45.825791 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.825291 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" event={"ID":"6886918f-844a-4d85-b4b6-5c389c5dd1f7","Type":"ContainerDied","Data":"2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76"} Jul 28 18:38:45.825791 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.825341 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4" event={"ID":"6886918f-844a-4d85-b4b6-5c389c5dd1f7","Type":"ContainerDied","Data":"271bfddbd957590da3cf9595bbfaecac372aaa9434acc991d5cc00f80a46f0f5"} Jul 28 18:38:45.825791 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.825363 2575 scope.go:117] "RemoveContainer" containerID="2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76" Jul 28 18:38:45.835242 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.835219 2575 scope.go:117] "RemoveContainer" containerID="43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976" Jul 28 18:38:45.844302 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.844280 2575 scope.go:117] "RemoveContainer" containerID="2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76" Jul 28 18:38:45.844566 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:38:45.844547 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76\": container with ID starting with 2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76 not found: ID does not exist" containerID="2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76" Jul 28 18:38:45.844616 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.844575 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76"} err="failed to get container status \"2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76\": rpc error: code = NotFound desc = could not find container \"2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76\": container with ID starting with 2fe3e7690759504b292e26e3624040365ba8b26d2396e158171a60e741c7ad76 not found: ID does not exist" Jul 28 18:38:45.844616 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.844593 2575 scope.go:117] "RemoveContainer" containerID="43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976" Jul 28 18:38:45.844860 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:38:45.844842 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976\": container with ID starting with 43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976 not found: ID does not exist" containerID="43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976" Jul 28 18:38:45.844903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.844866 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976"} err="failed to get container status \"43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976\": rpc error: code = NotFound desc = could not find container \"43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976\": container with ID starting with 43eb2df9bc7a649b925fa18a2125a1b15bdd761aa0af772d8c6389d9a0d7a976 not found: ID does not exist" Jul 28 18:38:45.848517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.848492 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4"] Jul 28 18:38:45.854423 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:45.854394 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-210335-openshift-default-7669d7c896-zzft4"] Jul 28 18:38:47.060463 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:47.060425 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" path="/var/lib/kubelet/pods/6886918f-844a-4d85-b4b6-5c389c5dd1f7/volumes" Jul 28 18:38:47.648138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:47.648105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:47.706732 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:47.706669 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:38:58.803920 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.803886 2575 patch_prober.go:28] interesting pod/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:48054->10.132.0.66:15021: read: connection reset by peer" start-of-body= Jul 28 18:38:58.804350 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.803954 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": read tcp 10.132.0.2:48054->10.132.0.66:15021: read: connection reset by peer" Jul 28 18:38:58.905898 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.905864 2575 generic.go:356] "Generic (PLEG): container finished" podID="74c55657-5059-47fb-9f0a-7c453b953797" containerID="e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb" exitCode=0 Jul 28 18:38:58.906087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.905936 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" event={"ID":"74c55657-5059-47fb-9f0a-7c453b953797","Type":"ContainerDied","Data":"e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb"} Jul 28 18:38:58.906354 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.906341 2575 scope.go:117] "RemoveContainer" containerID="e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb" Jul 28 18:38:58.907407 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.907376 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:38:58.907516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.907462 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:38:58.907516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:58.907508 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:38:59.914048 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:38:59.914010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" event={"ID":"74c55657-5059-47fb-9f0a-7c453b953797","Type":"ContainerStarted","Data":"6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67"} Jul 28 18:39:00.234427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:00.234316 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:39:00.234804 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:00.234773 2575 patch_prober.go:28] interesting pod/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 18:39:00.234915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:00.234832 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 18:39:01.235484 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:01.235432 2575 patch_prober.go:28] interesting pod/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 18:39:01.236008 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:01.235506 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 18:39:02.235426 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:02.235371 2575 patch_prober.go:28] interesting pod/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 18:39:02.235663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:02.235465 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 18:39:03.234709 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:03.234653 2575 patch_prober.go:28] interesting pod/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Jul 28 18:39:03.235017 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:03.234983 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Jul 28 18:39:04.237745 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:04.237708 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:39:04.238135 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:04.238063 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:39:04.238857 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:04.238835 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:39:08.864851 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:08.864816 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:09.337555 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.337522 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:09.649384 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.649336 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv"] Jul 28 18:39:09.649820 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.649804 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" Jul 28 18:39:09.649878 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.649821 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" Jul 28 18:39:09.649936 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.649927 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" Jul 28 18:39:09.649978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.649938 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" Jul 28 18:39:09.652154 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.652137 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.655086 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.655059 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-51a15d-openshift-default-dockercfg-7qrz4\"" Jul 28 18:39:09.665704 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.665653 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv"] Jul 28 18:39:09.680519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.680479 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:09.734003 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.733954 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-data\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734003 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734006 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-podinfo\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734243 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734025 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istiod-ca-cert\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734243 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734140 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psn8c\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-kube-api-access-psn8c\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734243 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734193 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-credential-socket\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734247 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-certs\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734272 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-envoy\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734331 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-token\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.734405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.734374 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-socket\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835049 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835006 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-credential-socket\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835257 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835060 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-certs\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835257 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835173 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-envoy\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835257 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835222 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-token\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835257 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835252 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-socket\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835337 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-data\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835383 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-podinfo\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835395 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-certs\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835410 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istiod-ca-cert\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835480 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835452 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-credential-socket\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835759 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835483 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psn8c\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-kube-api-access-psn8c\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835759 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835707 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-socket\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.835889 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.835766 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-data\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.836372 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.836344 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istiod-ca-cert\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.837698 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.837666 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-envoy\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.837934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.837914 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-podinfo\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.844920 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.844879 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-token\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.845077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.844919 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psn8c\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-kube-api-access-psn8c\") pod \"e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:09.897123 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.897086 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:09.964650 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:09.964552 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:10.118113 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:10.118086 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv"] Jul 28 18:39:10.120039 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:39:10.120004 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83b45a4d_e65c_49d0_bf3b_d91e0987720e.slice/crio-c2ba944117454bf507be17032ee3d2a12b16c0709e29a30fcfa93d4642f7e17f WatchSource:0}: Error finding container c2ba944117454bf507be17032ee3d2a12b16c0709e29a30fcfa93d4642f7e17f: Status 404 returned error can't find the container with id c2ba944117454bf507be17032ee3d2a12b16c0709e29a30fcfa93d4642f7e17f Jul 28 18:39:10.122396 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:10.122345 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:39:10.122508 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:10.122453 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:39:10.122553 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:10.122504 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:39:10.984041 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:10.984001 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" event={"ID":"83b45a4d-e65c-49d0-bf3b-d91e0987720e","Type":"ContainerStarted","Data":"a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd"} Jul 28 18:39:10.984437 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:10.984065 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" event={"ID":"83b45a4d-e65c-49d0-bf3b-d91e0987720e","Type":"ContainerStarted","Data":"c2ba944117454bf507be17032ee3d2a12b16c0709e29a30fcfa93d4642f7e17f"} Jul 28 18:39:11.007731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:11.007647 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podStartSLOduration=2.007626472 podStartE2EDuration="2.007626472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:11.005363035 +0000 UTC m=+2548.420020037" watchObservedRunningTime="2026-07-28 18:39:11.007626472 +0000 UTC m=+2548.422283475" Jul 28 18:39:11.965650 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:11.965609 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:11.973215 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:11.973180 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:11.989558 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:11.989523 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:11.990485 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:11.990467 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:12.964368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:12.964326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:13.152290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:13.152246 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:13.354023 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:13.353905 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:13.947196 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:13.947143 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:14.141915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:14.141873 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:14.243335 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:14.243234 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:27.130993 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:27.129601 2575 patch_prober.go:28] interesting pod/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": read tcp 10.132.0.2:58862->10.132.0.67:15021: read: connection reset by peer" start-of-body= Jul 28 18:39:27.130993 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:27.129664 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": read tcp 10.132.0.2:58862->10.132.0.67:15021: read: connection reset by peer" Jul 28 18:39:28.093297 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:28.093257 2575 generic.go:356] "Generic (PLEG): container finished" podID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerID="a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd" exitCode=0 Jul 28 18:39:28.093476 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:28.093335 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" event={"ID":"83b45a4d-e65c-49d0-bf3b-d91e0987720e","Type":"ContainerDied","Data":"a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd"} Jul 28 18:39:28.093777 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:28.093760 2575 scope.go:117] "RemoveContainer" containerID="a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd" Jul 28 18:39:28.094881 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:28.094849 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:39:28.094956 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:28.094940 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:39:28.094992 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:28.094980 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:39:29.101393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:29.101355 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" event={"ID":"83b45a4d-e65c-49d0-bf3b-d91e0987720e","Type":"ContainerStarted","Data":"395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8"} Jul 28 18:39:29.965322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:29.965276 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:29.965657 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:29.965628 2575 patch_prober.go:28] interesting pod/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 28 18:39:29.965821 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:29.965735 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 28 18:39:30.964964 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:30.964921 2575 patch_prober.go:28] interesting pod/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 28 18:39:30.965354 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:30.964982 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 28 18:39:31.965090 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:31.965045 2575 patch_prober.go:28] interesting pod/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 28 18:39:31.965505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:31.965120 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 28 18:39:32.965866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:32.965826 2575 patch_prober.go:28] interesting pod/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Jul 28 18:39:32.966250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:32.965904 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Jul 28 18:39:33.968845 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:33.968804 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:33.969233 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:33.969076 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:33.969783 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:33.969764 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:39:43.161069 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:43.161031 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:39:43.500125 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:43.499898 2575 scope.go:117] "RemoveContainer" containerID="327a76abf459fb062b259f302a4bf253a809cd66e394a0833973528d71f2c15f" Jul 28 18:39:46.337819 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.337777 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2"] Jul 28 18:39:46.338229 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.338156 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" Jul 28 18:39:46.338229 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.338167 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="6886918f-844a-4d85-b4b6-5c389c5dd1f7" containerName="istio-proxy" Jul 28 18:39:46.342418 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.342399 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:46.345018 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.344987 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:39:46.345139 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.345046 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:39:46.349377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.349347 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2"] Jul 28 18:39:46.389126 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.389078 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cec13b-0425-4df5-8d63-9d9f7b05a018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2\" (UID: \"55cec13b-0425-4df5-8d63-9d9f7b05a018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:46.489926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.489880 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cec13b-0425-4df5-8d63-9d9f7b05a018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2\" (UID: \"55cec13b-0425-4df5-8d63-9d9f7b05a018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:46.490526 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.490503 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cec13b-0425-4df5-8d63-9d9f7b05a018-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2\" (UID: \"55cec13b-0425-4df5-8d63-9d9f7b05a018\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:46.654586 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.654545 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:46.790880 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:46.790853 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2"] Jul 28 18:39:46.792154 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:39:46.792122 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55cec13b_0425_4df5_8d63_9d9f7b05a018.slice/crio-f599a235ce0fbfdf06c179ec6a678b63a38ce7cb7b9aea61ac159d93c6de3f99 WatchSource:0}: Error finding container f599a235ce0fbfdf06c179ec6a678b63a38ce7cb7b9aea61ac159d93c6de3f99: Status 404 returned error can't find the container with id f599a235ce0fbfdf06c179ec6a678b63a38ce7cb7b9aea61ac159d93c6de3f99 Jul 28 18:39:47.214878 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:47.214775 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" event={"ID":"55cec13b-0425-4df5-8d63-9d9f7b05a018","Type":"ContainerStarted","Data":"12368963abc17a664b7f333a33aae5af367bde7ec3529b443a8a823a74d5503f"} Jul 28 18:39:47.214878 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:47.214835 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" event={"ID":"55cec13b-0425-4df5-8d63-9d9f7b05a018","Type":"ContainerStarted","Data":"f599a235ce0fbfdf06c179ec6a678b63a38ce7cb7b9aea61ac159d93c6de3f99"} Jul 28 18:39:47.238465 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:47.238410 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" podStartSLOduration=1.238392423 podStartE2EDuration="1.238392423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:39:47.237150309 +0000 UTC m=+2584.651807312" watchObservedRunningTime="2026-07-28 18:39:47.238392423 +0000 UTC m=+2584.653049425" Jul 28 18:39:48.222093 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:48.222060 2575 generic.go:356] "Generic (PLEG): container finished" podID="55cec13b-0425-4df5-8d63-9d9f7b05a018" containerID="12368963abc17a664b7f333a33aae5af367bde7ec3529b443a8a823a74d5503f" exitCode=0 Jul 28 18:39:48.222576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:48.222148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" event={"ID":"55cec13b-0425-4df5-8d63-9d9f7b05a018","Type":"ContainerDied","Data":"12368963abc17a664b7f333a33aae5af367bde7ec3529b443a8a823a74d5503f"} Jul 28 18:39:49.367256 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:49.367222 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:49.417009 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:49.416968 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cec13b-0425-4df5-8d63-9d9f7b05a018-maas-api-service-ca\") pod \"55cec13b-0425-4df5-8d63-9d9f7b05a018\" (UID: \"55cec13b-0425-4df5-8d63-9d9f7b05a018\") " Jul 28 18:39:49.417445 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:49.417416 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55cec13b-0425-4df5-8d63-9d9f7b05a018-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55cec13b-0425-4df5-8d63-9d9f7b05a018" (UID: "55cec13b-0425-4df5-8d63-9d9f7b05a018"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:39:49.518262 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:49.518163 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55cec13b-0425-4df5-8d63-9d9f7b05a018-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:39:50.235979 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:50.235947 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" Jul 28 18:39:50.235979 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:50.235975 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2" event={"ID":"55cec13b-0425-4df5-8d63-9d9f7b05a018","Type":"ContainerDied","Data":"f599a235ce0fbfdf06c179ec6a678b63a38ce7cb7b9aea61ac159d93c6de3f99"} Jul 28 18:39:50.236237 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:39:50.236020 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f599a235ce0fbfdf06c179ec6a678b63a38ce7cb7b9aea61ac159d93c6de3f99" Jul 28 18:40:01.867055 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:01.867014 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:02.045709 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:02.045653 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:02.148630 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:02.148593 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:02.204184 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:02.204134 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2"] Jul 28 18:40:02.206664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:02.206628 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-a7357c-s8kc2"] Jul 28 18:40:02.224969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:02.224931 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:02.603463 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:02.603373 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:03.059172 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:03.059137 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55cec13b-0425-4df5-8d63-9d9f7b05a018" path="/var/lib/kubelet/pods/55cec13b-0425-4df5-8d63-9d9f7b05a018/volumes" Jul 28 18:40:08.778289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:08.778206 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:08.885425 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:08.885390 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:09.107243 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.107149 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg"] Jul 28 18:40:09.107462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.107430 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" containerID="cri-o://6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67" gracePeriod=30 Jul 28 18:40:09.134672 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.134638 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:09.368550 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.368455 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:09.494377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.494336 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457"] Jul 28 18:40:09.494843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.494822 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55cec13b-0425-4df5-8d63-9d9f7b05a018" containerName="revoke-keys" Jul 28 18:40:09.494949 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.494844 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="55cec13b-0425-4df5-8d63-9d9f7b05a018" containerName="revoke-keys" Jul 28 18:40:09.495014 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.494951 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="55cec13b-0425-4df5-8d63-9d9f7b05a018" containerName="revoke-keys" Jul 28 18:40:09.498355 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.498331 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:09.500977 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.500950 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:40:09.501103 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.501025 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:40:09.507534 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.507507 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457"] Jul 28 18:40:09.596762 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.596732 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bd6e6a-31bb-4139-b408-efc67c33cddf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-51a15d-fz457\" (UID: \"46bd6e6a-31bb-4139-b408-efc67c33cddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:09.697917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.697872 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bd6e6a-31bb-4139-b408-efc67c33cddf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-51a15d-fz457\" (UID: \"46bd6e6a-31bb-4139-b408-efc67c33cddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:09.698516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.698493 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bd6e6a-31bb-4139-b408-efc67c33cddf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-51a15d-fz457\" (UID: \"46bd6e6a-31bb-4139-b408-efc67c33cddf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:09.810049 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.810007 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:09.965853 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:09.965823 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457"] Jul 28 18:40:09.966298 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:40:09.966271 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46bd6e6a_31bb_4139_b408_efc67c33cddf.slice/crio-7a2ddcc4d3616e8fa04f51e36805b622e5780582d2dade04372f98b56fee36b5 WatchSource:0}: Error finding container 7a2ddcc4d3616e8fa04f51e36805b622e5780582d2dade04372f98b56fee36b5: Status 404 returned error can't find the container with id 7a2ddcc4d3616e8fa04f51e36805b622e5780582d2dade04372f98b56fee36b5 Jul 28 18:40:10.359507 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:10.359473 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" event={"ID":"46bd6e6a-31bb-4139-b408-efc67c33cddf","Type":"ContainerStarted","Data":"961c2331e88ec3c1561e0e1d2418a06988e609c7511d7c84faa4522ece907bcd"} Jul 28 18:40:10.359727 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:10.359529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" event={"ID":"46bd6e6a-31bb-4139-b408-efc67c33cddf","Type":"ContainerStarted","Data":"7a2ddcc4d3616e8fa04f51e36805b622e5780582d2dade04372f98b56fee36b5"} Jul 28 18:40:10.380337 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:10.380279 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" podStartSLOduration=1.380262877 podStartE2EDuration="1.380262877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:10.379437697 +0000 UTC m=+2607.794094701" watchObservedRunningTime="2026-07-28 18:40:10.380262877 +0000 UTC m=+2607.794919880" Jul 28 18:40:10.476559 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:10.476526 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:11.366452 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:11.366416 2575 generic.go:356] "Generic (PLEG): container finished" podID="46bd6e6a-31bb-4139-b408-efc67c33cddf" containerID="961c2331e88ec3c1561e0e1d2418a06988e609c7511d7c84faa4522ece907bcd" exitCode=0 Jul 28 18:40:11.366887 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:11.366502 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" event={"ID":"46bd6e6a-31bb-4139-b408-efc67c33cddf","Type":"ContainerDied","Data":"961c2331e88ec3c1561e0e1d2418a06988e609c7511d7c84faa4522ece907bcd"} Jul 28 18:40:12.510945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:12.510914 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:12.626151 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:12.626053 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bd6e6a-31bb-4139-b408-efc67c33cddf-maas-api-service-ca\") pod \"46bd6e6a-31bb-4139-b408-efc67c33cddf\" (UID: \"46bd6e6a-31bb-4139-b408-efc67c33cddf\") " Jul 28 18:40:12.626463 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:12.626438 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46bd6e6a-31bb-4139-b408-efc67c33cddf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46bd6e6a-31bb-4139-b408-efc67c33cddf" (UID: "46bd6e6a-31bb-4139-b408-efc67c33cddf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:40:12.727735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:12.727698 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46bd6e6a-31bb-4139-b408-efc67c33cddf-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:13.380667 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:13.380633 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" Jul 28 18:40:13.380894 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:13.380663 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457" event={"ID":"46bd6e6a-31bb-4139-b408-efc67c33cddf","Type":"ContainerDied","Data":"7a2ddcc4d3616e8fa04f51e36805b622e5780582d2dade04372f98b56fee36b5"} Jul 28 18:40:13.380894 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:13.380744 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a2ddcc4d3616e8fa04f51e36805b622e5780582d2dade04372f98b56fee36b5" Jul 28 18:40:14.372397 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.372332 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:40:14.387348 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.387318 2575 generic.go:356] "Generic (PLEG): container finished" podID="74c55657-5059-47fb-9f0a-7c453b953797" containerID="6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67" exitCode=0 Jul 28 18:40:14.387492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.387370 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" Jul 28 18:40:14.387492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.387403 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" event={"ID":"74c55657-5059-47fb-9f0a-7c453b953797","Type":"ContainerDied","Data":"6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67"} Jul 28 18:40:14.387492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.387459 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg" event={"ID":"74c55657-5059-47fb-9f0a-7c453b953797","Type":"ContainerDied","Data":"703ece3805be534608a05acbf6c785765d965f5b81a638bb525ede0bd842e21a"} Jul 28 18:40:14.387492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.387483 2575 scope.go:117] "RemoveContainer" containerID="6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67" Jul 28 18:40:14.398590 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.398567 2575 scope.go:117] "RemoveContainer" containerID="e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb" Jul 28 18:40:14.407568 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.407546 2575 scope.go:117] "RemoveContainer" containerID="6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67" Jul 28 18:40:14.407857 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:40:14.407834 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67\": container with ID starting with 6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67 not found: ID does not exist" containerID="6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67" Jul 28 18:40:14.407920 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.407869 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67"} err="failed to get container status \"6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67\": rpc error: code = NotFound desc = could not find container \"6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67\": container with ID starting with 6d5127ebffbc94a88cc9bc4dce4480257aaea378b8f36fbc5ccdbc35527f7b67 not found: ID does not exist" Jul 28 18:40:14.407920 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.407888 2575 scope.go:117] "RemoveContainer" containerID="e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb" Jul 28 18:40:14.408170 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:40:14.408144 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb\": container with ID starting with e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb not found: ID does not exist" containerID="e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb" Jul 28 18:40:14.408207 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.408181 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb"} err="failed to get container status \"e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb\": rpc error: code = NotFound desc = could not find container \"e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb\": container with ID starting with e1b6db1e05270fce0a146082aaf6538c8230113bf6492a3567a92b6aff2755eb not found: ID does not exist" Jul 28 18:40:14.548232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548198 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-envoy\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548241 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88lgv\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-kube-api-access-88lgv\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548278 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c55657-5059-47fb-9f0a-7c453b953797-istiod-ca-cert\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548334 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-credential-socket\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548360 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-istio-token\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548396 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-certs\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548422 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c55657-5059-47fb-9f0a-7c453b953797-istio-podinfo\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548462 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-socket\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548489 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-data\") pod \"74c55657-5059-47fb-9f0a-7c453b953797\" (UID: \"74c55657-5059-47fb-9f0a-7c453b953797\") " Jul 28 18:40:14.548753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548632 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:14.548971 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548759 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:14.548971 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548780 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74c55657-5059-47fb-9f0a-7c453b953797-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:40:14.549087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548978 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74c55657-5059-47fb-9f0a-7c453b953797-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.549087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548998 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.549087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.548999 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:14.549087 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.549014 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.549510 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.549485 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-data" (OuterVolumeSpecName: "istio-data") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:14.550648 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.550622 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-istio-token" (OuterVolumeSpecName: "istio-token") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:40:14.550742 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.550666 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-kube-api-access-88lgv" (OuterVolumeSpecName: "kube-api-access-88lgv") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "kube-api-access-88lgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:40:14.550879 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.550860 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:14.550971 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.550958 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74c55657-5059-47fb-9f0a-7c453b953797-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74c55657-5059-47fb-9f0a-7c453b953797" (UID: "74c55657-5059-47fb-9f0a-7c453b953797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:40:14.649780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.649742 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.649780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.649773 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74c55657-5059-47fb-9f0a-7c453b953797-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.649780 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.649784 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.650019 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.649793 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.650019 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.649801 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74c55657-5059-47fb-9f0a-7c453b953797-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.650019 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.649809 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88lgv\" (UniqueName: \"kubernetes.io/projected/74c55657-5059-47fb-9f0a-7c453b953797-kube-api-access-88lgv\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:14.712645 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.712596 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg"] Jul 28 18:40:14.718634 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:14.718600 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a7357c-openshift-default-7d4545f58d-bdjmg"] Jul 28 18:40:15.059509 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:15.059415 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74c55657-5059-47fb-9f0a-7c453b953797" path="/var/lib/kubelet/pods/74c55657-5059-47fb-9f0a-7c453b953797/volumes" Jul 28 18:40:25.121180 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.121146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:25.222889 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.222832 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:25.293085 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.293045 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457"] Jul 28 18:40:25.296948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.296916 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:25.297112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.296991 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-51a15d-fz457"] Jul 28 18:40:25.362212 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.362178 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:25.785598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:25.785562 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:27.059492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:27.059453 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46bd6e6a-31bb-4139-b408-efc67c33cddf" path="/var/lib/kubelet/pods/46bd6e6a-31bb-4139-b408-efc67c33cddf/volumes" Jul 28 18:40:31.728826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:31.728784 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:31.830139 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:31.830105 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:31.940908 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:31.940870 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv"] Jul 28 18:40:31.941161 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:31.941115 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" containerID="cri-o://395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8" gracePeriod=30 Jul 28 18:40:31.981274 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:31.981181 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:32.206567 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.206527 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:32.601335 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.601303 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:32.919119 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919078 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb"] Jul 28 18:40:32.919527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919490 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46bd6e6a-31bb-4139-b408-efc67c33cddf" containerName="revoke-keys" Jul 28 18:40:32.919527 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919502 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="46bd6e6a-31bb-4139-b408-efc67c33cddf" containerName="revoke-keys" Jul 28 18:40:32.919607 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919528 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" Jul 28 18:40:32.919607 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919534 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" Jul 28 18:40:32.919701 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919611 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="46bd6e6a-31bb-4139-b408-efc67c33cddf" containerName="revoke-keys" Jul 28 18:40:32.919701 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919625 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" Jul 28 18:40:32.919701 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.919634 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" Jul 28 18:40:32.922900 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.922881 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:32.925651 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.925626 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-5523b7-openshift-default-dockercfg-8zq5t\"" Jul 28 18:40:32.935723 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.935669 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb"] Jul 28 18:40:32.946426 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:32.946398 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:33.019735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.019652 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-credential-socket\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.019735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.019726 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-envoy\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.019772 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lz7j\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-kube-api-access-5lz7j\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.019866 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-certs\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.019916 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istiod-ca-cert\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.019966 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-token\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.020036 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-data\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020212 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.020074 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-podinfo\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.020212 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.020102 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-socket\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.121664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121629 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-certs\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.121664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121666 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istiod-ca-cert\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121712 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-token\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121749 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-data\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121778 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-podinfo\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121804 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-socket\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121849 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-credential-socket\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-envoy\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.121916 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lz7j\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-kube-api-access-5lz7j\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.122187 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-certs\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.122363 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-socket\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.122495 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-credential-socket\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.122520 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-data\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.122598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.122543 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istiod-ca-cert\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.124428 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.124406 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-envoy\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.124651 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.124631 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-podinfo\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.130913 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.130883 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-token\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.131096 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.131076 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lz7j\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-kube-api-access-5lz7j\") pod \"e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.234965 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.234868 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:33.384195 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.384160 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb"] Jul 28 18:40:33.384497 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:40:33.384470 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7bec57_4ab1_49eb_9279_0faaaed504b9.slice/crio-06383bec4bad582ee3e66e3a39a0735aa4bbbf1f21446fbdee6b2a526d8e0ba7 WatchSource:0}: Error finding container 06383bec4bad582ee3e66e3a39a0735aa4bbbf1f21446fbdee6b2a526d8e0ba7: Status 404 returned error can't find the container with id 06383bec4bad582ee3e66e3a39a0735aa4bbbf1f21446fbdee6b2a526d8e0ba7 Jul 28 18:40:33.386921 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.386873 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:40:33.387018 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.386973 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:40:33.387061 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.387019 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:40:33.498852 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.498740 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" event={"ID":"9a7bec57-4ab1-49eb-9279-0faaaed504b9","Type":"ContainerStarted","Data":"6112a5150ad990af759b5043df9ca86622283b1535e59db5cf7b7a9ed58a4bdb"} Jul 28 18:40:33.498852 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.498813 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" event={"ID":"9a7bec57-4ab1-49eb-9279-0faaaed504b9","Type":"ContainerStarted","Data":"06383bec4bad582ee3e66e3a39a0735aa4bbbf1f21446fbdee6b2a526d8e0ba7"} Jul 28 18:40:33.522627 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.522571 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podStartSLOduration=1.5225528499999998 podStartE2EDuration="1.52255285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:40:33.520268282 +0000 UTC m=+2630.934925286" watchObservedRunningTime="2026-07-28 18:40:33.52255285 +0000 UTC m=+2630.937209852" Jul 28 18:40:33.569841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.569809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:33.971417 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.971380 2575 patch_prober.go:28] interesting pod/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 28 18:40:33.971812 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:33.971440 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 28 18:40:34.235531 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:34.235433 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:34.251485 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:34.251449 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:34.504428 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:34.504337 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:34.505360 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:34.505338 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:35.518295 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:35.518266 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:35.631029 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:35.630999 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:35.733664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:35.733629 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:36.008204 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:36.008171 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:36.691972 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:36.691939 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:36.792798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:36.792760 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:40:37.199165 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.199136 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:40:37.259716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259586 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istiod-ca-cert\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.259716 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259634 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-token\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.259970 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259727 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-data\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.259970 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259869 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-podinfo\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.259970 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259901 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-certs\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.259970 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259938 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psn8c\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-kube-api-access-psn8c\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.260183 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.259992 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-credential-socket\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.260183 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260042 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-envoy\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.260183 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260143 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:37.260183 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260168 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:40:37.260385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260279 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-socket\") pod \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\" (UID: \"83b45a4d-e65c-49d0-bf3b-d91e0987720e\") " Jul 28 18:40:37.260385 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260289 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:37.260533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260448 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:37.260604 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260585 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-data" (OuterVolumeSpecName: "istio-data") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:37.260761 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260727 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.260761 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260749 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.260893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260771 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.260893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260785 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.260893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.260798 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.262258 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.262228 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:40:37.262455 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.262436 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:40:37.262518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.262457 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-token" (OuterVolumeSpecName: "istio-token") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:40:37.262518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.262487 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-kube-api-access-psn8c" (OuterVolumeSpecName: "kube-api-access-psn8c") pod "83b45a4d-e65c-49d0-bf3b-d91e0987720e" (UID: "83b45a4d-e65c-49d0-bf3b-d91e0987720e"). InnerVolumeSpecName "kube-api-access-psn8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:40:37.361461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.361418 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.361461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.361449 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psn8c\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-kube-api-access-psn8c\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.361461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.361462 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.361461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.361473 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83b45a4d-e65c-49d0-bf3b-d91e0987720e-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:40:37.530803 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.530721 2575 generic.go:356] "Generic (PLEG): container finished" podID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerID="395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8" exitCode=0 Jul 28 18:40:37.530948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.530820 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" event={"ID":"83b45a4d-e65c-49d0-bf3b-d91e0987720e","Type":"ContainerDied","Data":"395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8"} Jul 28 18:40:37.530948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.530862 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" Jul 28 18:40:37.530948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.530905 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv" event={"ID":"83b45a4d-e65c-49d0-bf3b-d91e0987720e","Type":"ContainerDied","Data":"c2ba944117454bf507be17032ee3d2a12b16c0709e29a30fcfa93d4642f7e17f"} Jul 28 18:40:37.530948 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.530936 2575 scope.go:117] "RemoveContainer" containerID="395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8" Jul 28 18:40:37.541021 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.540999 2575 scope.go:117] "RemoveContainer" containerID="a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd" Jul 28 18:40:37.550020 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.549995 2575 scope.go:117] "RemoveContainer" containerID="395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8" Jul 28 18:40:37.550323 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:40:37.550300 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8\": container with ID starting with 395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8 not found: ID does not exist" containerID="395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8" Jul 28 18:40:37.550378 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.550334 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8"} err="failed to get container status \"395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8\": rpc error: code = NotFound desc = could not find container \"395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8\": container with ID starting with 395e0aee7e14a82372371a5bc8aba0a52341256533830857f5e386758de1afc8 not found: ID does not exist" Jul 28 18:40:37.550378 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.550354 2575 scope.go:117] "RemoveContainer" containerID="a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd" Jul 28 18:40:37.550585 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:40:37.550563 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd\": container with ID starting with a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd not found: ID does not exist" containerID="a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd" Jul 28 18:40:37.550633 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.550591 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd"} err="failed to get container status \"a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd\": rpc error: code = NotFound desc = could not find container \"a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd\": container with ID starting with a3f32ec6c723682417d94e267438fd55ca432cc49b5959f4e99afd790065d7bd not found: ID does not exist" Jul 28 18:40:37.554337 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.554308 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv"] Jul 28 18:40:37.558092 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:37.558067 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-51a15d-openshift-default-7b68d47f44-pc4hv"] Jul 28 18:40:39.059890 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:39.059844 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" path="/var/lib/kubelet/pods/83b45a4d-e65c-49d0-bf3b-d91e0987720e/volumes" Jul 28 18:40:49.505794 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.505757 2575 patch_prober.go:28] interesting pod/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 18:40:49.506235 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.505819 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 18:40:49.604806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.604769 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerID="6112a5150ad990af759b5043df9ca86622283b1535e59db5cf7b7a9ed58a4bdb" exitCode=0 Jul 28 18:40:49.604992 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.604842 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" event={"ID":"9a7bec57-4ab1-49eb-9279-0faaaed504b9","Type":"ContainerDied","Data":"6112a5150ad990af759b5043df9ca86622283b1535e59db5cf7b7a9ed58a4bdb"} Jul 28 18:40:49.605293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.605275 2575 scope.go:117] "RemoveContainer" containerID="6112a5150ad990af759b5043df9ca86622283b1535e59db5cf7b7a9ed58a4bdb" Jul 28 18:40:49.606375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.606340 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:40:49.606495 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.606448 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:40:49.606565 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:49.606498 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:40:50.613077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:50.613036 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" event={"ID":"9a7bec57-4ab1-49eb-9279-0faaaed504b9","Type":"ContainerStarted","Data":"87db87915c843c42ce39dd4b69b652ce7bbc0575159dd2ba4e8e853b6f962edb"} Jul 28 18:40:51.235931 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:51.235896 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:51.236314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:51.236286 2575 patch_prober.go:28] interesting pod/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 18:40:51.236492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:51.236467 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 18:40:52.236746 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:52.236222 2575 patch_prober.go:28] interesting pod/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 18:40:52.236746 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:52.236309 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 18:40:53.236163 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:53.236114 2575 patch_prober.go:28] interesting pod/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 18:40:53.236386 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:53.236198 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 18:40:54.236056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:54.236010 2575 patch_prober.go:28] interesting pod/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Jul 28 18:40:54.236550 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:54.236092 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Jul 28 18:40:55.240108 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:55.240068 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:55.240562 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:55.240383 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:40:55.241098 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:40:55.241075 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:41:00.554739 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:00.554703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:00.902817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:00.902783 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:01.206730 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.206632 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5"] Jul 28 18:41:01.207224 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207208 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" Jul 28 18:41:01.207267 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207227 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="74c55657-5059-47fb-9f0a-7c453b953797" containerName="istio-proxy" Jul 28 18:41:01.207267 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207260 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" Jul 28 18:41:01.207334 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207269 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" Jul 28 18:41:01.207334 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207284 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" Jul 28 18:41:01.207334 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207292 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" Jul 28 18:41:01.207430 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207370 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" Jul 28 18:41:01.207430 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.207383 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="83b45a4d-e65c-49d0-bf3b-d91e0987720e" containerName="istio-proxy" Jul 28 18:41:01.210731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.210711 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.213607 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.213585 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-10fb8f-openshift-default-dockercfg-5vwh9\"" Jul 28 18:41:01.226085 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.226047 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5"] Jul 28 18:41:01.235971 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.235934 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:01.385106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385062 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-credential-socket\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385104 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istiod-ca-cert\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385183 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s7th\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-kube-api-access-2s7th\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385248 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-socket\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385269 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-envoy\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385365 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385344 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-certs\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385376 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-token\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385404 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-data\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.385520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.385466 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-podinfo\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.486782 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486651 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-envoy\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.486782 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486760 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-certs\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486796 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-token\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486830 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-data\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486873 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-podinfo\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486912 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-credential-socket\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486933 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istiod-ca-cert\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.486979 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s7th\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-kube-api-access-2s7th\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.487022 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-socket\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.487182 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-certs\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.487261 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-data\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487422 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.487367 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-socket\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487567 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.487458 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-credential-socket\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.487784 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.487767 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istiod-ca-cert\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.489146 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.489122 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-envoy\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.489699 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.489665 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-podinfo\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.495655 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.495612 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-token\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.495826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.495739 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s7th\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-kube-api-access-2s7th\") pod \"e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.521782 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.521743 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:01.681060 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.681029 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5"] Jul 28 18:41:01.682747 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:41:01.682712 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1b9351_0815_4bd8_8b06_b8933243e5fb.slice/crio-393d2ac6705ac734064c38304ffcfc23a19f3ff0b014978b7151645818c65dbd WatchSource:0}: Error finding container 393d2ac6705ac734064c38304ffcfc23a19f3ff0b014978b7151645818c65dbd: Status 404 returned error can't find the container with id 393d2ac6705ac734064c38304ffcfc23a19f3ff0b014978b7151645818c65dbd Jul 28 18:41:01.684964 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.684926 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:41:01.685068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.684998 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:41:01.685068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.685029 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:41:01.766129 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:01.766097 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:02.684212 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:02.684171 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" event={"ID":"8d1b9351-0815-4bd8-8b06-b8933243e5fb","Type":"ContainerStarted","Data":"76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b"} Jul 28 18:41:02.684678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:02.684232 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" event={"ID":"8d1b9351-0815-4bd8-8b06-b8933243e5fb","Type":"ContainerStarted","Data":"393d2ac6705ac734064c38304ffcfc23a19f3ff0b014978b7151645818c65dbd"} Jul 28 18:41:02.707914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:02.707852 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podStartSLOduration=1.707825063 podStartE2EDuration="1.707825063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:41:02.705668953 +0000 UTC m=+2660.120325955" watchObservedRunningTime="2026-07-28 18:41:02.707825063 +0000 UTC m=+2660.122482135" Jul 28 18:41:03.522238 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:03.522195 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:03.527452 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:03.527421 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:03.690450 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:03.690414 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:03.691525 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:03.691505 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:04.738071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:04.738038 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:04.833475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:04.833445 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:04.998392 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:04.998301 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:05.337004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:05.336921 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:05.570815 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:05.570781 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:05.652382 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:05.652347 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:19.731901 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:19.731854 2575 patch_prober.go:28] interesting pod/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 28 18:41:19.732555 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:19.731951 2575 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 28 18:41:20.794766 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:20.794727 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerID="76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b" exitCode=0 Jul 28 18:41:20.794766 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:20.794770 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" event={"ID":"8d1b9351-0815-4bd8-8b06-b8933243e5fb","Type":"ContainerDied","Data":"76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b"} Jul 28 18:41:20.795237 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:20.795167 2575 scope.go:117] "RemoveContainer" containerID="76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b" Jul 28 18:41:20.796238 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:20.796206 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:41:20.796292 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:20.796281 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:41:20.796327 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:20.796314 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:41:21.803010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:21.802971 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" event={"ID":"8d1b9351-0815-4bd8-8b06-b8933243e5fb","Type":"ContainerStarted","Data":"556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445"} Jul 28 18:41:22.522753 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:22.522714 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:22.523005 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:22.522981 2575 patch_prober.go:28] interesting pod/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 18:41:22.523062 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:22.523043 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 18:41:23.522638 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:23.522600 2575 patch_prober.go:28] interesting pod/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 18:41:23.523099 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:23.522656 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 18:41:24.523129 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:24.523086 2575 patch_prober.go:28] interesting pod/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 18:41:24.523545 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:24.523147 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 18:41:25.522420 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:25.522377 2575 patch_prober.go:28] interesting pod/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Jul 28 18:41:25.522612 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:25.522432 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Jul 28 18:41:26.526539 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:26.526504 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:26.526953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:26.526868 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:26.527579 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:26.527561 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:41:29.718630 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:29.718600 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:41:39.627272 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:41:39.627190 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:42:16.342204 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:42:16.342169 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:43:56.292612 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:43:56.292572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:44:43.595877 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:44:43.595796 2575 scope.go:117] "RemoveContainer" containerID="289a6c7b4fce3e8a8f9aeee4d30b95e1d32ee5c08df97787b17b9d054eba0117" Jul 28 18:44:43.604777 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:44:43.604756 2575 scope.go:117] "RemoveContainer" containerID="4e657f05feb0572adf5400ab655311be7d656b3c8f47892ea8f88d42067d76df" Jul 28 18:45:00.140653 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.140609 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq"] Jul 28 18:45:00.144110 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.144086 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr"] Jul 28 18:45:00.144250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.144227 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" Jul 28 18:45:00.146838 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.146811 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:45:00.147255 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.147224 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9"] Jul 28 18:45:00.147427 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.147407 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" Jul 28 18:45:00.150570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.150546 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" Jul 28 18:45:00.155987 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.155961 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" Jul 28 18:45:00.159020 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.158987 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr"] Jul 28 18:45:00.164981 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.163726 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq"] Jul 28 18:45:00.166574 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.166541 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9"] Jul 28 18:45:00.169918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.169888 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" Jul 28 18:45:00.202402 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.202372 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" Jul 28 18:45:00.390164 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.390135 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9"] Jul 28 18:45:00.391913 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:45:00.391843 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa95f59_dca6_4fa4_90de_6fac445426e5.slice/crio-6c0c9a1cf810b2a10c9ab8dc8cde0910fe5e5ebd5ae4a32cef0e1f286d0fe4d9 WatchSource:0}: Error finding container 6c0c9a1cf810b2a10c9ab8dc8cde0910fe5e5ebd5ae4a32cef0e1f286d0fe4d9: Status 404 returned error can't find the container with id 6c0c9a1cf810b2a10c9ab8dc8cde0910fe5e5ebd5ae4a32cef0e1f286d0fe4d9 Jul 28 18:45:00.547209 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.547176 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq"] Jul 28 18:45:00.548739 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:45:00.548709 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1980385_8dcb_450d_a8f8_71cf228b24b6.slice/crio-99355cecb1fdf127cb263777200e258daf3ff2d330eae9698414f004c36287db WatchSource:0}: Error finding container 99355cecb1fdf127cb263777200e258daf3ff2d330eae9698414f004c36287db: Status 404 returned error can't find the container with id 99355cecb1fdf127cb263777200e258daf3ff2d330eae9698414f004c36287db Jul 28 18:45:00.570867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:00.570841 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr"] Jul 28 18:45:00.581894 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:45:00.581860 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499d13f7_b9db_45f4_912e_10fcd98eb579.slice/crio-5cb10e34ebff51beadaf4f487ef8d554f40f309c7bcbf833e27e26e93aea934b WatchSource:0}: Error finding container 5cb10e34ebff51beadaf4f487ef8d554f40f309c7bcbf833e27e26e93aea934b: Status 404 returned error can't find the container with id 5cb10e34ebff51beadaf4f487ef8d554f40f309c7bcbf833e27e26e93aea934b Jul 28 18:45:01.080463 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.080423 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" event={"ID":"cfa95f59-dca6-4fa4-90de-6fac445426e5","Type":"ContainerStarted","Data":"2092e7a29e2f30068d2cdc5e4213b81ffd7a3ab751dcb093418af717d095bc19"} Jul 28 18:45:01.080660 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.080489 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" event={"ID":"cfa95f59-dca6-4fa4-90de-6fac445426e5","Type":"ContainerStarted","Data":"6c0c9a1cf810b2a10c9ab8dc8cde0910fe5e5ebd5ae4a32cef0e1f286d0fe4d9"} Jul 28 18:45:01.082031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.082000 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" event={"ID":"499d13f7-b9db-45f4-912e-10fcd98eb579","Type":"ContainerStarted","Data":"0b4bb3cdb03ca8323f570090cb40e92dc3c5e5830a13817df35bd0044c49cdfa"} Jul 28 18:45:01.082162 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.082054 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" event={"ID":"499d13f7-b9db-45f4-912e-10fcd98eb579","Type":"ContainerStarted","Data":"5cb10e34ebff51beadaf4f487ef8d554f40f309c7bcbf833e27e26e93aea934b"} Jul 28 18:45:01.083482 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.083457 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" event={"ID":"b1980385-8dcb-450d-a8f8-71cf228b24b6","Type":"ContainerStarted","Data":"67113c6ad40b044d9cc9a93cd8f546ce0dbd0591129570839c8bd6d5390beadf"} Jul 28 18:45:01.083599 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.083502 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" event={"ID":"b1980385-8dcb-450d-a8f8-71cf228b24b6","Type":"ContainerStarted","Data":"99355cecb1fdf127cb263777200e258daf3ff2d330eae9698414f004c36287db"} Jul 28 18:45:01.095377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.095324 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" podStartSLOduration=1.095306041 podStartE2EDuration="1.095306041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:45:01.094613808 +0000 UTC m=+2898.509270909" watchObservedRunningTime="2026-07-28 18:45:01.095306041 +0000 UTC m=+2898.509963044" Jul 28 18:45:01.110668 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.110614 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" podStartSLOduration=1.110600158 podStartE2EDuration="1.110600158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:45:01.10948034 +0000 UTC m=+2898.524137341" watchObservedRunningTime="2026-07-28 18:45:01.110600158 +0000 UTC m=+2898.525257161" Jul 28 18:45:01.126423 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:01.126366 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" podStartSLOduration=1.126349915 podStartE2EDuration="1.126349915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:45:01.123090892 +0000 UTC m=+2898.537747895" watchObservedRunningTime="2026-07-28 18:45:01.126349915 +0000 UTC m=+2898.541006954" Jul 28 18:45:02.091630 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:02.091597 2575 generic.go:356] "Generic (PLEG): container finished" podID="cfa95f59-dca6-4fa4-90de-6fac445426e5" containerID="2092e7a29e2f30068d2cdc5e4213b81ffd7a3ab751dcb093418af717d095bc19" exitCode=0 Jul 28 18:45:02.092103 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:02.091679 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" event={"ID":"cfa95f59-dca6-4fa4-90de-6fac445426e5","Type":"ContainerDied","Data":"2092e7a29e2f30068d2cdc5e4213b81ffd7a3ab751dcb093418af717d095bc19"} Jul 28 18:45:02.093141 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:02.093116 2575 generic.go:356] "Generic (PLEG): container finished" podID="499d13f7-b9db-45f4-912e-10fcd98eb579" containerID="0b4bb3cdb03ca8323f570090cb40e92dc3c5e5830a13817df35bd0044c49cdfa" exitCode=0 Jul 28 18:45:02.093248 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:02.093199 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" event={"ID":"499d13f7-b9db-45f4-912e-10fcd98eb579","Type":"ContainerDied","Data":"0b4bb3cdb03ca8323f570090cb40e92dc3c5e5830a13817df35bd0044c49cdfa"} Jul 28 18:45:02.094562 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:02.094543 2575 generic.go:356] "Generic (PLEG): container finished" podID="b1980385-8dcb-450d-a8f8-71cf228b24b6" containerID="67113c6ad40b044d9cc9a93cd8f546ce0dbd0591129570839c8bd6d5390beadf" exitCode=0 Jul 28 18:45:02.094648 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:02.094617 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" event={"ID":"b1980385-8dcb-450d-a8f8-71cf228b24b6","Type":"ContainerDied","Data":"67113c6ad40b044d9cc9a93cd8f546ce0dbd0591129570839c8bd6d5390beadf"} Jul 28 18:45:03.259316 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:03.259279 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" Jul 28 18:45:03.295595 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:03.295568 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" Jul 28 18:45:03.299943 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:03.299915 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" Jul 28 18:45:04.108936 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.108902 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" Jul 28 18:45:04.108936 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.108918 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr" event={"ID":"499d13f7-b9db-45f4-912e-10fcd98eb579","Type":"ContainerDied","Data":"5cb10e34ebff51beadaf4f487ef8d554f40f309c7bcbf833e27e26e93aea934b"} Jul 28 18:45:04.109177 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.108971 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb10e34ebff51beadaf4f487ef8d554f40f309c7bcbf833e27e26e93aea934b" Jul 28 18:45:04.110491 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.110461 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" Jul 28 18:45:04.110639 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.110490 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29754405-gjxbq" event={"ID":"b1980385-8dcb-450d-a8f8-71cf228b24b6","Type":"ContainerDied","Data":"99355cecb1fdf127cb263777200e258daf3ff2d330eae9698414f004c36287db"} Jul 28 18:45:04.110639 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.110528 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99355cecb1fdf127cb263777200e258daf3ff2d330eae9698414f004c36287db" Jul 28 18:45:04.112174 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.112148 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" event={"ID":"cfa95f59-dca6-4fa4-90de-6fac445426e5","Type":"ContainerDied","Data":"6c0c9a1cf810b2a10c9ab8dc8cde0910fe5e5ebd5ae4a32cef0e1f286d0fe4d9"} Jul 28 18:45:04.112174 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.112176 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9" Jul 28 18:45:04.112314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:04.112188 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c0c9a1cf810b2a10c9ab8dc8cde0910fe5e5ebd5ae4a32cef0e1f286d0fe4d9" Jul 28 18:45:14.353641 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:14.353610 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:45:15.613193 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:45:15.613157 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:46:04.829636 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:46:04.829553 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:46:43.640567 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:46:43.640531 2575 scope.go:117] "RemoveContainer" containerID="961c2331e88ec3c1561e0e1d2418a06988e609c7511d7c84faa4522ece907bcd" Jul 28 18:46:43.649924 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:46:43.649901 2575 scope.go:117] "RemoveContainer" containerID="12368963abc17a664b7f333a33aae5af367bde7ec3529b443a8a823a74d5503f" Jul 28 18:46:53.441775 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:46:53.441738 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:46:54.415007 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:46:54.414970 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:47:02.012127 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012086 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq"] Jul 28 18:47:02.012589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012533 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1980385-8dcb-450d-a8f8-71cf228b24b6" containerName="cleanup" Jul 28 18:47:02.012589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012548 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1980385-8dcb-450d-a8f8-71cf228b24b6" containerName="cleanup" Jul 28 18:47:02.012589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012568 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="499d13f7-b9db-45f4-912e-10fcd98eb579" containerName="cleanup" Jul 28 18:47:02.012589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012575 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="499d13f7-b9db-45f4-912e-10fcd98eb579" containerName="cleanup" Jul 28 18:47:02.012589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012588 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa95f59-dca6-4fa4-90de-6fac445426e5" containerName="cleanup" Jul 28 18:47:02.012806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012595 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa95f59-dca6-4fa4-90de-6fac445426e5" containerName="cleanup" Jul 28 18:47:02.012806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012667 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="499d13f7-b9db-45f4-912e-10fcd98eb579" containerName="cleanup" Jul 28 18:47:02.012806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012676 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1980385-8dcb-450d-a8f8-71cf228b24b6" containerName="cleanup" Jul 28 18:47:02.012806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.012698 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa95f59-dca6-4fa4-90de-6fac445426e5" containerName="cleanup" Jul 28 18:47:02.015903 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.015884 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.019701 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.019639 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5523b7\"/\"openshift-service-ca.crt\"" Jul 28 18:47:02.019882 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.019653 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5523b7\"/\"sub-test-model-5523b7-kserve-self-signed-certs\"" Jul 28 18:47:02.019882 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.019808 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-5523b7\"/\"default-dockercfg-d9r88\"" Jul 28 18:47:02.019882 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.019808 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-5523b7\"/\"kube-root-ca.crt\"" Jul 28 18:47:02.026844 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.026803 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq"] Jul 28 18:47:02.087891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.087845 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-dshm\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.087891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.087896 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-model-cache\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.088155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.087915 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-tmp-dir\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.088155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.087964 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9884e9c-14e7-49da-8e38-61f1389d258a-tls-certs\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.088155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.087987 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbw5r\" (UniqueName: \"kubernetes.io/projected/b9884e9c-14e7-49da-8e38-61f1389d258a-kube-api-access-kbw5r\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.088155 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.088037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-home\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.100736 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.100703 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:47:02.188518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.188475 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-dshm\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.188518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.188521 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-model-cache\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.188841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.188538 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-tmp-dir\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.188841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.188568 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9884e9c-14e7-49da-8e38-61f1389d258a-tls-certs\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.188841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.188584 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbw5r\" (UniqueName: \"kubernetes.io/projected/b9884e9c-14e7-49da-8e38-61f1389d258a-kube-api-access-kbw5r\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.188841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.188613 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-home\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.189060 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.189032 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-model-cache\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.189102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.189085 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-home\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.189102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.189092 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-tmp-dir\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.190974 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.190954 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-dshm\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.191179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.191160 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9884e9c-14e7-49da-8e38-61f1389d258a-tls-certs\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.197825 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.197795 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbw5r\" (UniqueName: \"kubernetes.io/projected/b9884e9c-14e7-49da-8e38-61f1389d258a-kube-api-access-kbw5r\") pod \"sub-test-model-5523b7-kserve-85487d787f-hp8cq\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.327335 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.327241 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:02.477625 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.477594 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq"] Jul 28 18:47:02.479768 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:47:02.479717 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9884e9c_14e7_49da_8e38_61f1389d258a.slice/crio-22d08bde349f755170d28a1c896dfa78571f62c9c17690510d23db55ee93e68e WatchSource:0}: Error finding container 22d08bde349f755170d28a1c896dfa78571f62c9c17690510d23db55ee93e68e: Status 404 returned error can't find the container with id 22d08bde349f755170d28a1c896dfa78571f62c9c17690510d23db55ee93e68e Jul 28 18:47:02.481965 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.481941 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:47:02.814783 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:02.814734 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" event={"ID":"b9884e9c-14e7-49da-8e38-61f1389d258a","Type":"ContainerStarted","Data":"22d08bde349f755170d28a1c896dfa78571f62c9c17690510d23db55ee93e68e"} Jul 28 18:47:03.822543 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:03.822506 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" event={"ID":"b9884e9c-14e7-49da-8e38-61f1389d258a","Type":"ContainerStarted","Data":"a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a"} Jul 28 18:47:03.842882 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:03.842816 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" podStartSLOduration=1.6901198979999998 podStartE2EDuration="2.842796204s" totalImagesPullingTime="1.152676306s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:47:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:47:03.839353526 +0000 UTC m=+3021.254010529" watchObservedRunningTime="2026-07-28 18:47:03.842796204 +0000 UTC m=+3021.257453207" Jul 28 18:47:12.327606 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:12.327562 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:12.328077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:12.327622 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:12.341876 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:12.341834 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:12.885647 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:12.885610 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:47:56.005156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:47:56.005079 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:48:07.138477 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:48:07.138439 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:48:35.936023 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:48:35.935987 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:49:14.824540 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:14.824456 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq"] Jul 28 18:49:14.843953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:14.843921 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:49:15.695554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:15.695518 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:49:24.792250 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:24.792216 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:49:29.844296 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:29.844230 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" podUID="b9884e9c-14e7-49da-8e38-61f1389d258a" containerName="main" containerID="cri-o://a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a" gracePeriod=45 Jul 28 18:49:30.118718 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.118670 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:49:30.159433 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159391 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbw5r\" (UniqueName: \"kubernetes.io/projected/b9884e9c-14e7-49da-8e38-61f1389d258a-kube-api-access-kbw5r\") pod \"b9884e9c-14e7-49da-8e38-61f1389d258a\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " Jul 28 18:49:30.159433 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159451 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-model-cache\") pod \"b9884e9c-14e7-49da-8e38-61f1389d258a\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " Jul 28 18:49:30.159712 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159475 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-tmp-dir\") pod \"b9884e9c-14e7-49da-8e38-61f1389d258a\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " Jul 28 18:49:30.159712 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159526 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-dshm\") pod \"b9884e9c-14e7-49da-8e38-61f1389d258a\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " Jul 28 18:49:30.159712 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159552 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9884e9c-14e7-49da-8e38-61f1389d258a-tls-certs\") pod \"b9884e9c-14e7-49da-8e38-61f1389d258a\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " Jul 28 18:49:30.159712 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-home\") pod \"b9884e9c-14e7-49da-8e38-61f1389d258a\" (UID: \"b9884e9c-14e7-49da-8e38-61f1389d258a\") " Jul 28 18:49:30.159963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159804 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b9884e9c-14e7-49da-8e38-61f1389d258a" (UID: "b9884e9c-14e7-49da-8e38-61f1389d258a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:30.159963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.159829 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b9884e9c-14e7-49da-8e38-61f1389d258a" (UID: "b9884e9c-14e7-49da-8e38-61f1389d258a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:30.160063 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.160000 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-home" (OuterVolumeSpecName: "home") pod "b9884e9c-14e7-49da-8e38-61f1389d258a" (UID: "b9884e9c-14e7-49da-8e38-61f1389d258a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:30.160063 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.160009 2575 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-model-cache\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:49:30.160063 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.160047 2575 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-tmp-dir\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:49:30.162003 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.161958 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-dshm" (OuterVolumeSpecName: "dshm") pod "b9884e9c-14e7-49da-8e38-61f1389d258a" (UID: "b9884e9c-14e7-49da-8e38-61f1389d258a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:49:30.162131 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.162027 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9884e9c-14e7-49da-8e38-61f1389d258a-kube-api-access-kbw5r" (OuterVolumeSpecName: "kube-api-access-kbw5r") pod "b9884e9c-14e7-49da-8e38-61f1389d258a" (UID: "b9884e9c-14e7-49da-8e38-61f1389d258a"). InnerVolumeSpecName "kube-api-access-kbw5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:49:30.162179 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.162134 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9884e9c-14e7-49da-8e38-61f1389d258a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b9884e9c-14e7-49da-8e38-61f1389d258a" (UID: "b9884e9c-14e7-49da-8e38-61f1389d258a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:49:30.260796 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.260754 2575 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-dshm\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:49:30.260796 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.260789 2575 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9884e9c-14e7-49da-8e38-61f1389d258a-tls-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:49:30.260796 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.260801 2575 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9884e9c-14e7-49da-8e38-61f1389d258a-home\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:49:30.261052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.260810 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbw5r\" (UniqueName: \"kubernetes.io/projected/b9884e9c-14e7-49da-8e38-61f1389d258a-kube-api-access-kbw5r\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:49:30.662248 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.662211 2575 generic.go:356] "Generic (PLEG): container finished" podID="b9884e9c-14e7-49da-8e38-61f1389d258a" containerID="a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a" exitCode=0 Jul 28 18:49:30.662468 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.662262 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" Jul 28 18:49:30.662468 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.662265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" event={"ID":"b9884e9c-14e7-49da-8e38-61f1389d258a","Type":"ContainerDied","Data":"a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a"} Jul 28 18:49:30.662468 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.662382 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq" event={"ID":"b9884e9c-14e7-49da-8e38-61f1389d258a","Type":"ContainerDied","Data":"22d08bde349f755170d28a1c896dfa78571f62c9c17690510d23db55ee93e68e"} Jul 28 18:49:30.662468 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.662407 2575 scope.go:117] "RemoveContainer" containerID="a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a" Jul 28 18:49:30.673180 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.673158 2575 scope.go:117] "RemoveContainer" containerID="a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a" Jul 28 18:49:30.673482 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:49:30.673459 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a\": container with ID starting with a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a not found: ID does not exist" containerID="a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a" Jul 28 18:49:30.673534 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.673492 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a"} err="failed to get container status \"a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a\": rpc error: code = NotFound desc = could not find container \"a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a\": container with ID starting with a8c171d2059196699f04e6e23317099036e50aad4076466a7f5f01590e7a1d4a not found: ID does not exist" Jul 28 18:49:30.688254 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.688213 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq"] Jul 28 18:49:30.692281 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:30.692246 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-5523b7/sub-test-model-5523b7-kserve-85487d787f-hp8cq"] Jul 28 18:49:31.059268 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:31.059179 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9884e9c-14e7-49da-8e38-61f1389d258a" path="/var/lib/kubelet/pods/b9884e9c-14e7-49da-8e38-61f1389d258a/volumes" Jul 28 18:49:45.934019 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:49:45.933976 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:50:56.209369 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:56.209280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:50:57.331321 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:57.331267 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:50:58.111814 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.111762 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-140048517608368"] Jul 28 18:50:58.112202 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.112188 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9884e9c-14e7-49da-8e38-61f1389d258a" containerName="main" Jul 28 18:50:58.112248 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.112203 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9884e9c-14e7-49da-8e38-61f1389d258a" containerName="main" Jul 28 18:50:58.112289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.112281 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9884e9c-14e7-49da-8e38-61f1389d258a" containerName="main" Jul 28 18:50:58.114337 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.114319 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:50:58.117589 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.117563 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-8785d\"" Jul 28 18:50:58.124931 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.124902 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517608368"] Jul 28 18:50:58.265600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.265556 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whdmp\" (UniqueName: \"kubernetes.io/projected/26215a67-449e-461a-a544-d74d495d63ac-kube-api-access-whdmp\") pod \"test-curl-7299-140048517608368\" (UID: \"26215a67-449e-461a-a544-d74d495d63ac\") " pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:50:58.366981 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.366876 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whdmp\" (UniqueName: \"kubernetes.io/projected/26215a67-449e-461a-a544-d74d495d63ac-kube-api-access-whdmp\") pod \"test-curl-7299-140048517608368\" (UID: \"26215a67-449e-461a-a544-d74d495d63ac\") " pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:50:58.375232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.375197 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whdmp\" (UniqueName: \"kubernetes.io/projected/26215a67-449e-461a-a544-d74d495d63ac-kube-api-access-whdmp\") pod \"test-curl-7299-140048517608368\" (UID: \"26215a67-449e-461a-a544-d74d495d63ac\") " pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:50:58.424316 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.424278 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:50:58.562829 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:58.562796 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517608368"] Jul 28 18:50:58.564580 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:50:58.564555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod26215a67_449e_461a_a544_d74d495d63ac.slice/crio-bb2351ba8d374d4a960b597072a6e35bbcae6e4af41c374eef417cdfa11b847a WatchSource:0}: Error finding container bb2351ba8d374d4a960b597072a6e35bbcae6e4af41c374eef417cdfa11b847a: Status 404 returned error can't find the container with id bb2351ba8d374d4a960b597072a6e35bbcae6e4af41c374eef417cdfa11b847a Jul 28 18:50:59.175620 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:50:59.175582 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517608368" event={"ID":"26215a67-449e-461a-a544-d74d495d63ac","Type":"ContainerStarted","Data":"bb2351ba8d374d4a960b597072a6e35bbcae6e4af41c374eef417cdfa11b847a"} Jul 28 18:51:00.185077 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:00.185042 2575 generic.go:356] "Generic (PLEG): container finished" podID="26215a67-449e-461a-a544-d74d495d63ac" containerID="b9a903e8bda776b24fe2ee10f3de5efae9f7e8d384c1e1b4e2b00d4805b60f23" exitCode=0 Jul 28 18:51:00.185499 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:00.185127 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517608368" event={"ID":"26215a67-449e-461a-a544-d74d495d63ac","Type":"ContainerDied","Data":"b9a903e8bda776b24fe2ee10f3de5efae9f7e8d384c1e1b4e2b00d4805b60f23"} Jul 28 18:51:01.329908 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.329878 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:51:01.349561 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.349528 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-140048517608368_26215a67-449e-461a-a544-d74d495d63ac/test-curl-7299-140048517608368/0.log" Jul 28 18:51:01.377341 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.377298 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-140048517608368"] Jul 28 18:51:01.381442 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.381409 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-140048517608368"] Jul 28 18:51:01.496834 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.496729 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whdmp\" (UniqueName: \"kubernetes.io/projected/26215a67-449e-461a-a544-d74d495d63ac-kube-api-access-whdmp\") pod \"26215a67-449e-461a-a544-d74d495d63ac\" (UID: \"26215a67-449e-461a-a544-d74d495d63ac\") " Jul 28 18:51:01.499040 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.499006 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26215a67-449e-461a-a544-d74d495d63ac-kube-api-access-whdmp" (OuterVolumeSpecName: "kube-api-access-whdmp") pod "26215a67-449e-461a-a544-d74d495d63ac" (UID: "26215a67-449e-461a-a544-d74d495d63ac"). InnerVolumeSpecName "kube-api-access-whdmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:01.517225 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.517186 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-140048517640880"] Jul 28 18:51:01.517620 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.517606 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26215a67-449e-461a-a544-d74d495d63ac" containerName="test-curl-7299-140048517608368" Jul 28 18:51:01.517664 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.517623 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="26215a67-449e-461a-a544-d74d495d63ac" containerName="test-curl-7299-140048517608368" Jul 28 18:51:01.517750 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.517740 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="26215a67-449e-461a-a544-d74d495d63ac" containerName="test-curl-7299-140048517608368" Jul 28 18:51:01.520199 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.520179 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:01.531694 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.531649 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517640880"] Jul 28 18:51:01.598485 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.598441 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whdmp\" (UniqueName: \"kubernetes.io/projected/26215a67-449e-461a-a544-d74d495d63ac-kube-api-access-whdmp\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:01.699575 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.699530 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k8vx\" (UniqueName: \"kubernetes.io/projected/2b1d01c8-f350-48b3-9ef8-99d4f2179f40-kube-api-access-9k8vx\") pod \"test-curl-7299-140048517640880\" (UID: \"2b1d01c8-f350-48b3-9ef8-99d4f2179f40\") " pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:01.800932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.800833 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k8vx\" (UniqueName: \"kubernetes.io/projected/2b1d01c8-f350-48b3-9ef8-99d4f2179f40-kube-api-access-9k8vx\") pod \"test-curl-7299-140048517640880\" (UID: \"2b1d01c8-f350-48b3-9ef8-99d4f2179f40\") " pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:01.816457 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.816418 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k8vx\" (UniqueName: \"kubernetes.io/projected/2b1d01c8-f350-48b3-9ef8-99d4f2179f40-kube-api-access-9k8vx\") pod \"test-curl-7299-140048517640880\" (UID: \"2b1d01c8-f350-48b3-9ef8-99d4f2179f40\") " pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:01.830362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.830323 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:01.975091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:01.975057 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517640880"] Jul 28 18:51:01.976380 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:01.976354 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b1d01c8_f350_48b3_9ef8_99d4f2179f40.slice/crio-85892a9f3553d040d312cd94f847cfe82ccca7b1928ae647629a7abc18660d71 WatchSource:0}: Error finding container 85892a9f3553d040d312cd94f847cfe82ccca7b1928ae647629a7abc18660d71: Status 404 returned error can't find the container with id 85892a9f3553d040d312cd94f847cfe82ccca7b1928ae647629a7abc18660d71 Jul 28 18:51:02.198210 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:02.198157 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517640880" event={"ID":"2b1d01c8-f350-48b3-9ef8-99d4f2179f40","Type":"ContainerStarted","Data":"85892a9f3553d040d312cd94f847cfe82ccca7b1928ae647629a7abc18660d71"} Jul 28 18:51:02.199497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:02.199478 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb2351ba8d374d4a960b597072a6e35bbcae6e4af41c374eef417cdfa11b847a" Jul 28 18:51:02.199623 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:02.199497 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:51:02.215469 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:51:02.215411 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7299-140048517608368\" is forbidden: User \"system:node:ip-10-0-135-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-147.ec2.internal' and this object" podUID="26215a67-449e-461a-a544-d74d495d63ac" pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:51:03.058137 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:51:03.058090 2575 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7299-140048517608368\" is forbidden: User \"system:node:ip-10-0-135-147.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-135-147.ec2.internal' and this object" podUID="26215a67-449e-461a-a544-d74d495d63ac" pod="opendatahub/test-curl-7299-140048517608368" Jul 28 18:51:03.059513 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:03.059484 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26215a67-449e-461a-a544-d74d495d63ac" path="/var/lib/kubelet/pods/26215a67-449e-461a-a544-d74d495d63ac/volumes" Jul 28 18:51:03.206913 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:03.206875 2575 generic.go:356] "Generic (PLEG): container finished" podID="2b1d01c8-f350-48b3-9ef8-99d4f2179f40" containerID="48d235ac6c68777ebe772b189e96e2ff2c1d1ab47a5c37f01fb34566c48910b0" exitCode=0 Jul 28 18:51:03.207102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:03.206958 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517640880" event={"ID":"2b1d01c8-f350-48b3-9ef8-99d4f2179f40","Type":"ContainerDied","Data":"48d235ac6c68777ebe772b189e96e2ff2c1d1ab47a5c37f01fb34566c48910b0"} Jul 28 18:51:04.347261 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.347231 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:04.367022 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.366996 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-140048517640880_2b1d01c8-f350-48b3-9ef8-99d4f2179f40/test-curl-7299-140048517640880/0.log" Jul 28 18:51:04.393942 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.393905 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-140048517640880"] Jul 28 18:51:04.396432 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.396404 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-140048517640880"] Jul 28 18:51:04.422398 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.422361 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k8vx\" (UniqueName: \"kubernetes.io/projected/2b1d01c8-f350-48b3-9ef8-99d4f2179f40-kube-api-access-9k8vx\") pod \"2b1d01c8-f350-48b3-9ef8-99d4f2179f40\" (UID: \"2b1d01c8-f350-48b3-9ef8-99d4f2179f40\") " Jul 28 18:51:04.424502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.424476 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1d01c8-f350-48b3-9ef8-99d4f2179f40-kube-api-access-9k8vx" (OuterVolumeSpecName: "kube-api-access-9k8vx") pod "2b1d01c8-f350-48b3-9ef8-99d4f2179f40" (UID: "2b1d01c8-f350-48b3-9ef8-99d4f2179f40"). InnerVolumeSpecName "kube-api-access-9k8vx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:04.523821 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.523670 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k8vx\" (UniqueName: \"kubernetes.io/projected/2b1d01c8-f350-48b3-9ef8-99d4f2179f40-kube-api-access-9k8vx\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:04.560105 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.560062 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:04.560477 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.560464 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b1d01c8-f350-48b3-9ef8-99d4f2179f40" containerName="test-curl-7299-140048517640880" Jul 28 18:51:04.560518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.560479 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b1d01c8-f350-48b3-9ef8-99d4f2179f40" containerName="test-curl-7299-140048517640880" Jul 28 18:51:04.560554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.560540 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b1d01c8-f350-48b3-9ef8-99d4f2179f40" containerName="test-curl-7299-140048517640880" Jul 28 18:51:04.562622 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.562605 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:04.568626 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.568558 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:04.625046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.625008 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w92c\" (UniqueName: \"kubernetes.io/projected/22ec1f9f-b194-4dff-a588-eef5b1b612b4-kube-api-access-7w92c\") pod \"test-curl-7299-140048517572624\" (UID: \"22ec1f9f-b194-4dff-a588-eef5b1b612b4\") " pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:04.725802 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.725742 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w92c\" (UniqueName: \"kubernetes.io/projected/22ec1f9f-b194-4dff-a588-eef5b1b612b4-kube-api-access-7w92c\") pod \"test-curl-7299-140048517572624\" (UID: \"22ec1f9f-b194-4dff-a588-eef5b1b612b4\") " pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:04.734444 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.734409 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w92c\" (UniqueName: \"kubernetes.io/projected/22ec1f9f-b194-4dff-a588-eef5b1b612b4-kube-api-access-7w92c\") pod \"test-curl-7299-140048517572624\" (UID: \"22ec1f9f-b194-4dff-a588-eef5b1b612b4\") " pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:04.872961 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:04.872876 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:05.010997 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:05.010966 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:05.012789 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:05.012763 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22ec1f9f_b194_4dff_a588_eef5b1b612b4.slice/crio-706b919bbdb6b6aae00ed77951b71aea748e3e59f6ab5db67871b570c15b4ca7 WatchSource:0}: Error finding container 706b919bbdb6b6aae00ed77951b71aea748e3e59f6ab5db67871b570c15b4ca7: Status 404 returned error can't find the container with id 706b919bbdb6b6aae00ed77951b71aea748e3e59f6ab5db67871b570c15b4ca7 Jul 28 18:51:05.059534 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:05.059498 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b1d01c8-f350-48b3-9ef8-99d4f2179f40" path="/var/lib/kubelet/pods/2b1d01c8-f350-48b3-9ef8-99d4f2179f40/volumes" Jul 28 18:51:05.219979 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:05.219940 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517640880" Jul 28 18:51:05.220182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:05.219971 2575 scope.go:117] "RemoveContainer" containerID="48d235ac6c68777ebe772b189e96e2ff2c1d1ab47a5c37f01fb34566c48910b0" Jul 28 18:51:05.221462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:05.221434 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517572624" event={"ID":"22ec1f9f-b194-4dff-a588-eef5b1b612b4","Type":"ContainerStarted","Data":"706b919bbdb6b6aae00ed77951b71aea748e3e59f6ab5db67871b570c15b4ca7"} Jul 28 18:51:06.229553 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:06.229519 2575 generic.go:356] "Generic (PLEG): container finished" podID="22ec1f9f-b194-4dff-a588-eef5b1b612b4" containerID="91220a0fb0126d2283879a2dfae088f0a8455e2795871bc4d8955216b1f4ced8" exitCode=0 Jul 28 18:51:06.230056 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:06.229614 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517572624" event={"ID":"22ec1f9f-b194-4dff-a588-eef5b1b612b4","Type":"ContainerDied","Data":"91220a0fb0126d2283879a2dfae088f0a8455e2795871bc4d8955216b1f4ced8"} Jul 28 18:51:07.370608 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.370580 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:07.389934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.389898 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-140048517572624_22ec1f9f-b194-4dff-a588-eef5b1b612b4/test-curl-7299-140048517572624/0.log" Jul 28 18:51:07.415093 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.415056 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:07.419189 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.419154 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:07.450786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.450751 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w92c\" (UniqueName: \"kubernetes.io/projected/22ec1f9f-b194-4dff-a588-eef5b1b612b4-kube-api-access-7w92c\") pod \"22ec1f9f-b194-4dff-a588-eef5b1b612b4\" (UID: \"22ec1f9f-b194-4dff-a588-eef5b1b612b4\") " Jul 28 18:51:07.452975 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.452943 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22ec1f9f-b194-4dff-a588-eef5b1b612b4-kube-api-access-7w92c" (OuterVolumeSpecName: "kube-api-access-7w92c") pod "22ec1f9f-b194-4dff-a588-eef5b1b612b4" (UID: "22ec1f9f-b194-4dff-a588-eef5b1b612b4"). InnerVolumeSpecName "kube-api-access-7w92c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:07.538464 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.538372 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:07.538835 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.538819 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22ec1f9f-b194-4dff-a588-eef5b1b612b4" containerName="test-curl-7299-140048517572624" Jul 28 18:51:07.538904 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.538837 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="22ec1f9f-b194-4dff-a588-eef5b1b612b4" containerName="test-curl-7299-140048517572624" Jul 28 18:51:07.538941 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.538915 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="22ec1f9f-b194-4dff-a588-eef5b1b612b4" containerName="test-curl-7299-140048517572624" Jul 28 18:51:07.540929 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.540914 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:07.549242 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.549214 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:07.551516 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.551489 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pbht\" (UniqueName: \"kubernetes.io/projected/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5-kube-api-access-4pbht\") pod \"test-curl-7299-140048517572624\" (UID: \"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5\") " pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:07.551636 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.551590 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w92c\" (UniqueName: \"kubernetes.io/projected/22ec1f9f-b194-4dff-a588-eef5b1b612b4-kube-api-access-7w92c\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:07.652426 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.652388 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pbht\" (UniqueName: \"kubernetes.io/projected/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5-kube-api-access-4pbht\") pod \"test-curl-7299-140048517572624\" (UID: \"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5\") " pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:07.660621 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.660590 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pbht\" (UniqueName: \"kubernetes.io/projected/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5-kube-api-access-4pbht\") pod \"test-curl-7299-140048517572624\" (UID: \"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5\") " pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:07.852338 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.852231 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:07.990830 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:07.990803 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:07.992255 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:07.992226 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28ee9aba_1ee6_4d6e_b4f4_ee4394dde5f5.slice/crio-3afb3dc0d284d845de78b22109feb8daee07386255b1f760d79c1284ab0097d2 WatchSource:0}: Error finding container 3afb3dc0d284d845de78b22109feb8daee07386255b1f760d79c1284ab0097d2: Status 404 returned error can't find the container with id 3afb3dc0d284d845de78b22109feb8daee07386255b1f760d79c1284ab0097d2 Jul 28 18:51:08.244349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:08.244308 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517572624" event={"ID":"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5","Type":"ContainerStarted","Data":"3afb3dc0d284d845de78b22109feb8daee07386255b1f760d79c1284ab0097d2"} Jul 28 18:51:08.245548 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:08.245525 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="706b919bbdb6b6aae00ed77951b71aea748e3e59f6ab5db67871b570c15b4ca7" Jul 28 18:51:08.245672 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:08.245558 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:08.258143 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:08.258107 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7299-140048517572624" oldPodUID="22ec1f9f-b194-4dff-a588-eef5b1b612b4" podUID="28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" Jul 28 18:51:09.059518 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:09.059478 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22ec1f9f-b194-4dff-a588-eef5b1b612b4" path="/var/lib/kubelet/pods/22ec1f9f-b194-4dff-a588-eef5b1b612b4/volumes" Jul 28 18:51:09.253136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:09.253098 2575 generic.go:356] "Generic (PLEG): container finished" podID="28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" containerID="e8ea3ae802e07f58a1b46e6e6ea5d25a0fb17c608bed936a40f2adcba77d6f73" exitCode=0 Jul 28 18:51:09.253314 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:09.253187 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517572624" event={"ID":"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5","Type":"ContainerDied","Data":"e8ea3ae802e07f58a1b46e6e6ea5d25a0fb17c608bed936a40f2adcba77d6f73"} Jul 28 18:51:10.396083 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.396055 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:10.417220 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.417191 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-140048517572624_28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5/test-curl-7299-140048517572624/0.log" Jul 28 18:51:10.442416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.442380 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:10.446536 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.446506 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-140048517572624"] Jul 28 18:51:10.479125 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.479091 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pbht\" (UniqueName: \"kubernetes.io/projected/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5-kube-api-access-4pbht\") pod \"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5\" (UID: \"28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5\") " Jul 28 18:51:10.481175 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.481142 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5-kube-api-access-4pbht" (OuterVolumeSpecName: "kube-api-access-4pbht") pod "28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" (UID: "28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5"). InnerVolumeSpecName "kube-api-access-4pbht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:10.578364 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.578283 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:10.578724 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.578710 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" containerName="test-curl-7299-140048517572624" Jul 28 18:51:10.578765 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.578726 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" containerName="test-curl-7299-140048517572624" Jul 28 18:51:10.578819 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.578810 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" containerName="test-curl-7299-140048517572624" Jul 28 18:51:10.580008 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.579985 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pbht\" (UniqueName: \"kubernetes.io/projected/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5-kube-api-access-4pbht\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:10.580771 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.580749 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:10.586213 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.586181 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:10.681224 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.681179 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnwhp\" (UniqueName: \"kubernetes.io/projected/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3-kube-api-access-gnwhp\") pod \"test-curl-7299-140048517573632\" (UID: \"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3\") " pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:10.781842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.781799 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnwhp\" (UniqueName: \"kubernetes.io/projected/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3-kube-api-access-gnwhp\") pod \"test-curl-7299-140048517573632\" (UID: \"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3\") " pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:10.790138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.790104 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnwhp\" (UniqueName: \"kubernetes.io/projected/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3-kube-api-access-gnwhp\") pod \"test-curl-7299-140048517573632\" (UID: \"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3\") " pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:10.891574 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:10.891538 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:11.020826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:11.020800 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:11.022479 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:11.022451 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bf7dad6_0e35_4fff_b03a_763e0b0ac8e3.slice/crio-aea9fad277967169216b52c20dcbd9c402a4063bcc97a87b96f99bf97cf9a281 WatchSource:0}: Error finding container aea9fad277967169216b52c20dcbd9c402a4063bcc97a87b96f99bf97cf9a281: Status 404 returned error can't find the container with id aea9fad277967169216b52c20dcbd9c402a4063bcc97a87b96f99bf97cf9a281 Jul 28 18:51:11.059802 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:11.059765 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5" path="/var/lib/kubelet/pods/28ee9aba-1ee6-4d6e-b4f4-ee4394dde5f5/volumes" Jul 28 18:51:11.266533 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:11.266431 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517573632" event={"ID":"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3","Type":"ContainerStarted","Data":"aea9fad277967169216b52c20dcbd9c402a4063bcc97a87b96f99bf97cf9a281"} Jul 28 18:51:11.267833 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:11.267815 2575 scope.go:117] "RemoveContainer" containerID="e8ea3ae802e07f58a1b46e6e6ea5d25a0fb17c608bed936a40f2adcba77d6f73" Jul 28 18:51:11.267918 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:11.267816 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517572624" Jul 28 18:51:12.275735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:12.275703 2575 generic.go:356] "Generic (PLEG): container finished" podID="1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" containerID="ae78e0fb3fe8405000194dbef1fa1f7bc45d3142fa41ebf457de5afc38011dd8" exitCode=0 Jul 28 18:51:12.276197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:12.275784 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517573632" event={"ID":"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3","Type":"ContainerDied","Data":"ae78e0fb3fe8405000194dbef1fa1f7bc45d3142fa41ebf457de5afc38011dd8"} Jul 28 18:51:13.424840 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.424812 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:13.445400 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.445365 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-140048517573632_1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3/test-curl-7299-140048517573632/0.log" Jul 28 18:51:13.472703 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.472640 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:13.475257 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.475232 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:13.507232 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.507192 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnwhp\" (UniqueName: \"kubernetes.io/projected/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3-kube-api-access-gnwhp\") pod \"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3\" (UID: \"1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3\") " Jul 28 18:51:13.509368 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.509342 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3-kube-api-access-gnwhp" (OuterVolumeSpecName: "kube-api-access-gnwhp") pod "1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" (UID: "1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3"). InnerVolumeSpecName "kube-api-access-gnwhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:13.601891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.601804 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:13.602197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.602184 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" containerName="test-curl-7299-140048517573632" Jul 28 18:51:13.602237 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.602199 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" containerName="test-curl-7299-140048517573632" Jul 28 18:51:13.602278 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.602270 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" containerName="test-curl-7299-140048517573632" Jul 28 18:51:13.604193 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.604176 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:13.608299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.608273 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnwhp\" (UniqueName: \"kubernetes.io/projected/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3-kube-api-access-gnwhp\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:13.615182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.615146 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:13.709461 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.709412 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcqbw\" (UniqueName: \"kubernetes.io/projected/82c483c0-a25c-4783-bce2-bb2735c2bed8-kube-api-access-bcqbw\") pod \"test-curl-7299-140048517573632\" (UID: \"82c483c0-a25c-4783-bce2-bb2735c2bed8\") " pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:13.810548 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.810505 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcqbw\" (UniqueName: \"kubernetes.io/projected/82c483c0-a25c-4783-bce2-bb2735c2bed8-kube-api-access-bcqbw\") pod \"test-curl-7299-140048517573632\" (UID: \"82c483c0-a25c-4783-bce2-bb2735c2bed8\") " pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:13.826665 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.826628 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcqbw\" (UniqueName: \"kubernetes.io/projected/82c483c0-a25c-4783-bce2-bb2735c2bed8-kube-api-access-bcqbw\") pod \"test-curl-7299-140048517573632\" (UID: \"82c483c0-a25c-4783-bce2-bb2735c2bed8\") " pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:13.915757 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:13.915715 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:14.053210 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.053144 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:14.054698 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:14.054660 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod82c483c0_a25c_4783_bce2_bb2735c2bed8.slice/crio-8b6a5266908747ecc441f547721b93cce10cadbe20c8dc1ef458f78ee64a6c9d WatchSource:0}: Error finding container 8b6a5266908747ecc441f547721b93cce10cadbe20c8dc1ef458f78ee64a6c9d: Status 404 returned error can't find the container with id 8b6a5266908747ecc441f547721b93cce10cadbe20c8dc1ef458f78ee64a6c9d Jul 28 18:51:14.062708 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.062666 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r98s"] Jul 28 18:51:14.065798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.065776 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:14.068408 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.068388 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-whsg2\"" Jul 28 18:51:14.075482 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.075459 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r98s"] Jul 28 18:51:14.114707 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.114640 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm5ql\" (UniqueName: \"kubernetes.io/projected/52af282d-a87a-42b7-ad2a-5d1756506920-kube-api-access-vm5ql\") pod \"kuadrant-operator-catalog-9r98s\" (UID: \"52af282d-a87a-42b7-ad2a-5d1756506920\") " pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:14.215579 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.215486 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm5ql\" (UniqueName: \"kubernetes.io/projected/52af282d-a87a-42b7-ad2a-5d1756506920-kube-api-access-vm5ql\") pod \"kuadrant-operator-catalog-9r98s\" (UID: \"52af282d-a87a-42b7-ad2a-5d1756506920\") " pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:14.224426 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.224394 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm5ql\" (UniqueName: \"kubernetes.io/projected/52af282d-a87a-42b7-ad2a-5d1756506920-kube-api-access-vm5ql\") pod \"kuadrant-operator-catalog-9r98s\" (UID: \"52af282d-a87a-42b7-ad2a-5d1756506920\") " pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:14.292002 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.291973 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aea9fad277967169216b52c20dcbd9c402a4063bcc97a87b96f99bf97cf9a281" Jul 28 18:51:14.292002 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.291989 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:14.293291 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.293265 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517573632" event={"ID":"82c483c0-a25c-4783-bce2-bb2735c2bed8","Type":"ContainerStarted","Data":"8b6a5266908747ecc441f547721b93cce10cadbe20c8dc1ef458f78ee64a6c9d"} Jul 28 18:51:14.304294 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.304264 2575 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7299-140048517573632" oldPodUID="1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" podUID="82c483c0-a25c-4783-bce2-bb2735c2bed8" Jul 28 18:51:14.377620 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.377594 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:14.517038 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:14.517003 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r98s"] Jul 28 18:51:14.518459 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:14.518431 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52af282d_a87a_42b7_ad2a_5d1756506920.slice/crio-5cfb4eb88942ac0473133921bbc6dcb451bc1c417afbebf97995aeff1606b782 WatchSource:0}: Error finding container 5cfb4eb88942ac0473133921bbc6dcb451bc1c417afbebf97995aeff1606b782: Status 404 returned error can't find the container with id 5cfb4eb88942ac0473133921bbc6dcb451bc1c417afbebf97995aeff1606b782 Jul 28 18:51:15.059054 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:15.059018 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3" path="/var/lib/kubelet/pods/1bf7dad6-0e35-4fff-b03a-763e0b0ac8e3/volumes" Jul 28 18:51:15.303261 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:15.303217 2575 generic.go:356] "Generic (PLEG): container finished" podID="82c483c0-a25c-4783-bce2-bb2735c2bed8" containerID="d748993d48c30ea89020247374d7726e531b26d23abc004269456a74e4406931" exitCode=0 Jul 28 18:51:15.303456 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:15.303440 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7299-140048517573632" event={"ID":"82c483c0-a25c-4783-bce2-bb2735c2bed8","Type":"ContainerDied","Data":"d748993d48c30ea89020247374d7726e531b26d23abc004269456a74e4406931"} Jul 28 18:51:15.305648 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:15.305607 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" event={"ID":"52af282d-a87a-42b7-ad2a-5d1756506920","Type":"ContainerStarted","Data":"5cfb4eb88942ac0473133921bbc6dcb451bc1c417afbebf97995aeff1606b782"} Jul 28 18:51:16.455610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.455584 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:16.479904 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.479871 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7299-140048517573632_82c483c0-a25c-4783-bce2-bb2735c2bed8/test-curl-7299-140048517573632/0.log" Jul 28 18:51:16.505639 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.505601 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:16.515945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.515396 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7299-140048517573632"] Jul 28 18:51:16.536747 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.536710 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcqbw\" (UniqueName: \"kubernetes.io/projected/82c483c0-a25c-4783-bce2-bb2735c2bed8-kube-api-access-bcqbw\") pod \"82c483c0-a25c-4783-bce2-bb2735c2bed8\" (UID: \"82c483c0-a25c-4783-bce2-bb2735c2bed8\") " Jul 28 18:51:16.538926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.538901 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c483c0-a25c-4783-bce2-bb2735c2bed8-kube-api-access-bcqbw" (OuterVolumeSpecName: "kube-api-access-bcqbw") pod "82c483c0-a25c-4783-bce2-bb2735c2bed8" (UID: "82c483c0-a25c-4783-bce2-bb2735c2bed8"). InnerVolumeSpecName "kube-api-access-bcqbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:16.638172 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:16.638134 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcqbw\" (UniqueName: \"kubernetes.io/projected/82c483c0-a25c-4783-bce2-bb2735c2bed8-kube-api-access-bcqbw\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:17.027348 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.027245 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:17.060169 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.060132 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c483c0-a25c-4783-bce2-bb2735c2bed8" path="/var/lib/kubelet/pods/82c483c0-a25c-4783-bce2-bb2735c2bed8/volumes" Jul 28 18:51:17.320398 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.320300 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" event={"ID":"52af282d-a87a-42b7-ad2a-5d1756506920","Type":"ContainerStarted","Data":"1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db"} Jul 28 18:51:17.321529 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.321505 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7299-140048517573632" Jul 28 18:51:17.321646 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.321561 2575 scope.go:117] "RemoveContainer" containerID="d748993d48c30ea89020247374d7726e531b26d23abc004269456a74e4406931" Jul 28 18:51:17.342923 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.342862 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" podStartSLOduration=1.491214214 podStartE2EDuration="3.342839453s" totalImagesPullingTime="1.851625239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:17.34120233 +0000 UTC m=+3274.755859338" watchObservedRunningTime="2026-07-28 18:51:17.342839453 +0000 UTC m=+3274.757496458" Jul 28 18:51:17.358486 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.358451 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9"] Jul 28 18:51:17.358875 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.358862 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c483c0-a25c-4783-bce2-bb2735c2bed8" containerName="test-curl-7299-140048517573632" Jul 28 18:51:17.358934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.358877 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c483c0-a25c-4783-bce2-bb2735c2bed8" containerName="test-curl-7299-140048517573632" Jul 28 18:51:17.358968 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.358946 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="82c483c0-a25c-4783-bce2-bb2735c2bed8" containerName="test-curl-7299-140048517573632" Jul 28 18:51:17.361980 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.361963 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.364656 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.364634 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-473596-openshift-default-dockercfg-fn8sc\"" Jul 28 18:51:17.378618 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.378589 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9"] Jul 28 18:51:17.387393 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.387356 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:17.445607 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445545 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-podinfo\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445662 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-credential-socket\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445744 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-socket\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445773 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-envoy\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445839 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x29m\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-kube-api-access-7x29m\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445867 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-certs\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445888 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istiod-ca-cert\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445923 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-token\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.445969 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.445972 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-data\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546380 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-token\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546425 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-data\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546449 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-podinfo\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546546 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-credential-socket\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546601 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-socket\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546626 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-envoy\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546704 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x29m\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-kube-api-access-7x29m\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546741 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-certs\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.546893 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546769 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istiod-ca-cert\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.547271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.546980 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-data\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.547271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.547054 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-socket\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.547271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.547093 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-credential-socket\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.547271 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.547179 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-certs\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.547411 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.547366 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istiod-ca-cert\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.548962 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.548940 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-envoy\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.549233 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.549214 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-podinfo\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.555649 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.555620 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-token\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.555798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.555735 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x29m\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-kube-api-access-7x29m\") pod \"e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.594459 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.594364 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:17.672735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.672674 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:17.822963 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.817335 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9"] Jul 28 18:51:17.824976 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.824933 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:51:17.825064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.825019 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:51:17.825064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:17.825057 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:51:18.329565 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:18.329528 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" event={"ID":"4ea639b1-f46f-4ed6-8bef-f1dde540e607","Type":"ContainerStarted","Data":"734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6"} Jul 28 18:51:18.329795 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:18.329585 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" event={"ID":"4ea639b1-f46f-4ed6-8bef-f1dde540e607","Type":"ContainerStarted","Data":"e193f3c57d8d1e7f34b1ed09aa716c7f7178e71fc72dcb95c267cb70d44b470f"} Jul 28 18:51:18.352361 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:18.352286 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" podStartSLOduration=1.352268593 podStartE2EDuration="1.352268593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:18.350121008 +0000 UTC m=+3275.764778010" watchObservedRunningTime="2026-07-28 18:51:18.352268593 +0000 UTC m=+3275.766925596" Jul 28 18:51:18.673769 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:18.673734 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:18.679124 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:18.679091 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:19.336048 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:19.336013 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:19.337078 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:19.337053 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:19.760361 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:19.760326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:20.348965 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:20.348929 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:20.523395 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:20.523361 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:20.625645 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:20.625529 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:20.824928 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:20.824896 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:21.185871 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:21.185833 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:21.289610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:21.289574 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:24.378534 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:24.378479 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:24.378978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:24.378554 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:24.401000 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:24.400972 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:25.391440 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:25.391413 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:25.425360 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:25.425318 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r98s"] Jul 28 18:51:27.381585 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:27.381542 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" podUID="52af282d-a87a-42b7-ad2a-5d1756506920" containerName="registry-server" containerID="cri-o://1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db" gracePeriod=2 Jul 28 18:51:27.653359 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:27.653330 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:27.745788 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:27.745745 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vm5ql\" (UniqueName: \"kubernetes.io/projected/52af282d-a87a-42b7-ad2a-5d1756506920-kube-api-access-vm5ql\") pod \"52af282d-a87a-42b7-ad2a-5d1756506920\" (UID: \"52af282d-a87a-42b7-ad2a-5d1756506920\") " Jul 28 18:51:27.747921 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:27.747889 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52af282d-a87a-42b7-ad2a-5d1756506920-kube-api-access-vm5ql" (OuterVolumeSpecName: "kube-api-access-vm5ql") pod "52af282d-a87a-42b7-ad2a-5d1756506920" (UID: "52af282d-a87a-42b7-ad2a-5d1756506920"). InnerVolumeSpecName "kube-api-access-vm5ql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:51:27.847282 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:27.847242 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm5ql\" (UniqueName: \"kubernetes.io/projected/52af282d-a87a-42b7-ad2a-5d1756506920-kube-api-access-vm5ql\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:51:28.389460 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.389425 2575 generic.go:356] "Generic (PLEG): container finished" podID="52af282d-a87a-42b7-ad2a-5d1756506920" containerID="1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db" exitCode=0 Jul 28 18:51:28.389905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.389489 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" Jul 28 18:51:28.389905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.389510 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" event={"ID":"52af282d-a87a-42b7-ad2a-5d1756506920","Type":"ContainerDied","Data":"1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db"} Jul 28 18:51:28.389905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.389579 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-9r98s" event={"ID":"52af282d-a87a-42b7-ad2a-5d1756506920","Type":"ContainerDied","Data":"5cfb4eb88942ac0473133921bbc6dcb451bc1c417afbebf97995aeff1606b782"} Jul 28 18:51:28.389905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.389601 2575 scope.go:117] "RemoveContainer" containerID="1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db" Jul 28 18:51:28.401866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.401839 2575 scope.go:117] "RemoveContainer" containerID="1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db" Jul 28 18:51:28.402307 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:51:28.402289 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db\": container with ID starting with 1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db not found: ID does not exist" containerID="1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db" Jul 28 18:51:28.402374 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.402316 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db"} err="failed to get container status \"1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db\": rpc error: code = NotFound desc = could not find container \"1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db\": container with ID starting with 1744b1392a308c6d7af9cc45ac6c3889c8c4394a3f9c81ad5ba61f6c7a9d65db not found: ID does not exist" Jul 28 18:51:28.415050 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.415010 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r98s"] Jul 28 18:51:28.419324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:28.419287 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-9r98s"] Jul 28 18:51:29.059517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:29.059433 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52af282d-a87a-42b7-ad2a-5d1756506920" path="/var/lib/kubelet/pods/52af282d-a87a-42b7-ad2a-5d1756506920/volumes" Jul 28 18:51:37.446757 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:37.446717 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerID="734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6" exitCode=0 Jul 28 18:51:37.447170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:37.446791 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" event={"ID":"4ea639b1-f46f-4ed6-8bef-f1dde540e607","Type":"ContainerDied","Data":"734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6"} Jul 28 18:51:37.447222 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:37.447174 2575 scope.go:117] "RemoveContainer" containerID="734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6" Jul 28 18:51:37.448224 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:37.448192 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:51:37.448327 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:37.448275 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:51:37.448449 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:37.448323 2575 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 28 18:51:38.461039 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:38.460997 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" event={"ID":"4ea639b1-f46f-4ed6-8bef-f1dde540e607","Type":"ContainerStarted","Data":"97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6"} Jul 28 18:51:38.673091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:38.673046 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:38.673342 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:38.673314 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 28 18:51:38.673402 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:38.673381 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 28 18:51:39.674010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:39.673966 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 28 18:51:39.674424 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:39.674028 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 28 18:51:40.674030 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:40.673983 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 28 18:51:40.674434 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:40.674049 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 28 18:51:41.673908 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:41.673866 2575 patch_prober.go:28] interesting pod/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Jul 28 18:51:41.674102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:41.673923 2575 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Jul 28 18:51:42.676958 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:42.676927 2575 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:42.677409 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:42.677384 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:42.685003 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:42.684975 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:51:45.331313 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:45.331257 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:46.030013 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:46.029969 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:46.354083 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:46.353986 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:46.525646 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:46.525616 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:49.318085 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.318046 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:49.412036 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.411996 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn"] Jul 28 18:51:49.412535 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.412515 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52af282d-a87a-42b7-ad2a-5d1756506920" containerName="registry-server" Jul 28 18:51:49.412535 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.412534 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="52af282d-a87a-42b7-ad2a-5d1756506920" containerName="registry-server" Jul 28 18:51:49.412773 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.412629 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="52af282d-a87a-42b7-ad2a-5d1756506920" containerName="registry-server" Jul 28 18:51:49.413190 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.413162 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:49.417498 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.417476 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.420116 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.420077 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:51:49.420116 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.420103 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5b063e-serving-cert\"" Jul 28 18:51:49.420293 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.420184 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 28 18:51:49.421106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.421087 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-b4b87\"" Jul 28 18:51:49.428178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.428147 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn"] Jul 28 18:51:49.494917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.494883 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:49.554998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.554963 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.555205 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.555037 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1181c218-a2b0-405b-9889-53ed1e913afc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.555205 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.555077 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf6vq\" (UniqueName: \"kubernetes.io/projected/1181c218-a2b0-405b-9889-53ed1e913afc-kube-api-access-xf6vq\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.655817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.655768 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf6vq\" (UniqueName: \"kubernetes.io/projected/1181c218-a2b0-405b-9889-53ed1e913afc-kube-api-access-xf6vq\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.656037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.655867 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.656037 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.655923 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1181c218-a2b0-405b-9889-53ed1e913afc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.656037 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:51:49.656018 2575 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-serving-cert: secret "maas-api-e2e-ipp-b-5b063e-serving-cert" not found Jul 28 18:51:49.656217 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:51:49.656114 2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls podName:1181c218-a2b0-405b-9889-53ed1e913afc nodeName:}" failed. No retries permitted until 2026-07-28 18:51:50.156090908 +0000 UTC m=+3307.570747889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls") pod "maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" (UID: "1181c218-a2b0-405b-9889-53ed1e913afc") : secret "maas-api-e2e-ipp-b-5b063e-serving-cert" not found Jul 28 18:51:49.656563 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.656544 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1181c218-a2b0-405b-9889-53ed1e913afc-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.677769 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.677738 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf6vq\" (UniqueName: \"kubernetes.io/projected/1181c218-a2b0-405b-9889-53ed1e913afc-kube-api-access-xf6vq\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:49.781731 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:49.781660 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:50.092279 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.092187 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:50.161002 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.160960 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:50.163503 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.163475 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:50.190591 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.190560 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:51:50.331492 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.331449 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:50.482242 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.482209 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn"] Jul 28 18:51:50.483923 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:51:50.483885 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1181c218_a2b0_405b_9889_53ed1e913afc.slice/crio-b7a3ceae45d84dfa97cb67ad4129ad94ecaf3288a2735c2201563f5a41e2a79b WatchSource:0}: Error finding container b7a3ceae45d84dfa97cb67ad4129ad94ecaf3288a2735c2201563f5a41e2a79b: Status 404 returned error can't find the container with id b7a3ceae45d84dfa97cb67ad4129ad94ecaf3288a2735c2201563f5a41e2a79b Jul 28 18:51:50.542495 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:50.542433 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" event={"ID":"1181c218-a2b0-405b-9889-53ed1e913afc","Type":"ContainerStarted","Data":"b7a3ceae45d84dfa97cb67ad4129ad94ecaf3288a2735c2201563f5a41e2a79b"} Jul 28 18:51:53.564868 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:53.564819 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" event={"ID":"1181c218-a2b0-405b-9889-53ed1e913afc","Type":"ContainerStarted","Data":"7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4"} Jul 28 18:51:53.565331 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:53.564921 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:51:53.582082 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:53.582024 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" podStartSLOduration=2.400531683 podStartE2EDuration="4.582004818s" totalImagesPullingTime="2.181473135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:51:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:51:53.581087381 +0000 UTC m=+3310.995744383" watchObservedRunningTime="2026-07-28 18:51:53.582004818 +0000 UTC m=+3310.996661819" Jul 28 18:51:59.576797 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:51:59.576763 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:52:15.006836 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:52:15.006742 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:52:21.257964 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:52:21.257928 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tprs8_43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c/payload-processing/0.log" Jul 28 18:52:35.731301 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:52:35.731260 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:16.602532 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:16.602472 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tprs8_43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c/payload-processing/0.log" Jul 28 18:53:17.042051 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:17.042003 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:17.375112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:17.375021 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:17.477130 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:17.477092 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:20.809073 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.809032 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:20.903977 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.903941 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k"] Jul 28 18:53:20.909968 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.909918 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:20.914672 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.914643 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:20.917617 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.917591 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-94ebbf-serving-cert\"" Jul 28 18:53:20.921866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.921840 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k"] Jul 28 18:53:20.990890 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.990824 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzgs8\" (UniqueName: \"kubernetes.io/projected/481a9d84-3a98-4398-895b-41ac5f683a11-kube-api-access-hzgs8\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:20.991095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.991046 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481a9d84-3a98-4398-895b-41ac5f683a11-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:20.991172 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:20.991119 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481a9d84-3a98-4398-895b-41ac5f683a11-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.048357 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.048323 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:21.092079 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.091990 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481a9d84-3a98-4398-895b-41ac5f683a11-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.092079 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.092063 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481a9d84-3a98-4398-895b-41ac5f683a11-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.092315 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.092148 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzgs8\" (UniqueName: \"kubernetes.io/projected/481a9d84-3a98-4398-895b-41ac5f683a11-kube-api-access-hzgs8\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.092975 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.092948 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481a9d84-3a98-4398-895b-41ac5f683a11-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.095136 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.095108 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481a9d84-3a98-4398-895b-41ac5f683a11-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.130088 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.130047 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzgs8\" (UniqueName: \"kubernetes.io/projected/481a9d84-3a98-4398-895b-41ac5f683a11-kube-api-access-hzgs8\") pod \"maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.224663 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.224609 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:21.225801 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.225774 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:21.377198 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.377168 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k"] Jul 28 18:53:21.378739 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:53:21.378708 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481a9d84_3a98_4398_895b_41ac5f683a11.slice/crio-7931858bc6854ea203d5dfb0f2106a8d16d03f3219980f3d006d77af4c864f08 WatchSource:0}: Error finding container 7931858bc6854ea203d5dfb0f2106a8d16d03f3219980f3d006d77af4c864f08: Status 404 returned error can't find the container with id 7931858bc6854ea203d5dfb0f2106a8d16d03f3219980f3d006d77af4c864f08 Jul 28 18:53:21.380142 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.380125 2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 18:53:21.548362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.548326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:21.635610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.635572 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:21.680850 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.680803 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd"] Jul 28 18:53:21.685529 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.685505 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.688573 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.688549 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-94ebbf-dockercfg-kc7kf\"" Jul 28 18:53:21.688867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.688849 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-94ebbf\"" Jul 28 18:53:21.696371 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.696342 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd"] Jul 28 18:53:21.698044 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.698018 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c4fc6-878e-4383-905d-ca0a56104f9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.698165 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.698054 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4stqn\" (UniqueName: \"kubernetes.io/projected/3f8c4fc6-878e-4383-905d-ca0a56104f9c-kube-api-access-4stqn\") pod \"payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.799497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.799453 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c4fc6-878e-4383-905d-ca0a56104f9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.799497 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.799499 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4stqn\" (UniqueName: \"kubernetes.io/projected/3f8c4fc6-878e-4383-905d-ca0a56104f9c-kube-api-access-4stqn\") pod \"payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.800205 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.800181 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c4fc6-878e-4383-905d-ca0a56104f9c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.812571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.812539 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4stqn\" (UniqueName: \"kubernetes.io/projected/3f8c4fc6-878e-4383-905d-ca0a56104f9c-kube-api-access-4stqn\") pod \"payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:21.998317 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:21.998205 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:22.103205 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:22.103165 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" event={"ID":"481a9d84-3a98-4398-895b-41ac5f683a11","Type":"ContainerStarted","Data":"30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599"} Jul 28 18:53:22.103396 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:22.103237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" event={"ID":"481a9d84-3a98-4398-895b-41ac5f683a11","Type":"ContainerStarted","Data":"7931858bc6854ea203d5dfb0f2106a8d16d03f3219980f3d006d77af4c864f08"} Jul 28 18:53:22.103473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:22.103389 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:22.123798 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:22.123717 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" podStartSLOduration=1.688500247 podStartE2EDuration="2.123692825s" totalImagesPullingTime="435.192578ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:22.121308977 +0000 UTC m=+3399.535965981" watchObservedRunningTime="2026-07-28 18:53:22.123692825 +0000 UTC m=+3399.538349820" Jul 28 18:53:22.151045 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:22.151007 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd"] Jul 28 18:53:22.154335 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:53:22.154305 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f8c4fc6_878e_4383_905d_ca0a56104f9c.slice/crio-bcd8cac77e8a4438c3735b9d0d26b06963373d281ae2a271aeffee5d53ad2f7a WatchSource:0}: Error finding container bcd8cac77e8a4438c3735b9d0d26b06963373d281ae2a271aeffee5d53ad2f7a: Status 404 returned error can't find the container with id bcd8cac77e8a4438c3735b9d0d26b06963373d281ae2a271aeffee5d53ad2f7a Jul 28 18:53:23.111272 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:23.111228 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" event={"ID":"3f8c4fc6-878e-4383-905d-ca0a56104f9c","Type":"ContainerStarted","Data":"7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56"} Jul 28 18:53:23.111696 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:23.111289 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" event={"ID":"3f8c4fc6-878e-4383-905d-ca0a56104f9c","Type":"ContainerStarted","Data":"bcd8cac77e8a4438c3735b9d0d26b06963373d281ae2a271aeffee5d53ad2f7a"} Jul 28 18:53:23.127469 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:23.127418 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" podStartSLOduration=2.127402126 podStartE2EDuration="2.127402126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:23.126652424 +0000 UTC m=+3400.541309426" watchObservedRunningTime="2026-07-28 18:53:23.127402126 +0000 UTC m=+3400.542059127" Jul 28 18:53:24.117201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:24.117161 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:28.117022 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:28.116988 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:35.125401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:35.125307 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:45.815000 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:45.814961 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:46.508789 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.508751 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872"] Jul 28 18:53:46.512878 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.512849 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:46.515392 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.515365 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:53:46.519761 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.519724 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872"] Jul 28 18:53:46.635550 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.635508 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d91a9f7-a1aa-416f-b442-0565d0ec3a63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872\" (UID: \"8d91a9f7-a1aa-416f-b442-0565d0ec3a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:46.736968 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.736926 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d91a9f7-a1aa-416f-b442-0565d0ec3a63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872\" (UID: \"8d91a9f7-a1aa-416f-b442-0565d0ec3a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:46.737637 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.737603 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d91a9f7-a1aa-416f-b442-0565d0ec3a63-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872\" (UID: \"8d91a9f7-a1aa-416f-b442-0565d0ec3a63\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:46.825576 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.825478 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:46.967147 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:46.967116 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872"] Jul 28 18:53:46.969650 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:53:46.969621 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d91a9f7_a1aa_416f_b442_0565d0ec3a63.slice/crio-35999670d132117b17fa5c83152fbe2c4cf86a0dae7a4d85f190092a4abfe24c WatchSource:0}: Error finding container 35999670d132117b17fa5c83152fbe2c4cf86a0dae7a4d85f190092a4abfe24c: Status 404 returned error can't find the container with id 35999670d132117b17fa5c83152fbe2c4cf86a0dae7a4d85f190092a4abfe24c Jul 28 18:53:47.258987 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:47.258951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" event={"ID":"8d91a9f7-a1aa-416f-b442-0565d0ec3a63","Type":"ContainerStarted","Data":"618d48b68b9ce73741178494ff97b1d7b21acf897c81a780b1c078d1e1cef199"} Jul 28 18:53:47.259137 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:47.259009 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" event={"ID":"8d91a9f7-a1aa-416f-b442-0565d0ec3a63","Type":"ContainerStarted","Data":"35999670d132117b17fa5c83152fbe2c4cf86a0dae7a4d85f190092a4abfe24c"} Jul 28 18:53:47.276572 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:47.276512 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" podStartSLOduration=1.276493407 podStartE2EDuration="1.276493407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:53:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:53:47.274907374 +0000 UTC m=+3424.689564377" watchObservedRunningTime="2026-07-28 18:53:47.276493407 +0000 UTC m=+3424.691150452" Jul 28 18:53:48.266610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:48.266573 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d91a9f7-a1aa-416f-b442-0565d0ec3a63" containerID="618d48b68b9ce73741178494ff97b1d7b21acf897c81a780b1c078d1e1cef199" exitCode=0 Jul 28 18:53:48.267124 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:48.266621 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" event={"ID":"8d91a9f7-a1aa-416f-b442-0565d0ec3a63","Type":"ContainerDied","Data":"618d48b68b9ce73741178494ff97b1d7b21acf897c81a780b1c078d1e1cef199"} Jul 28 18:53:49.415711 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:49.415654 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:49.565386 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:49.565281 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d91a9f7-a1aa-416f-b442-0565d0ec3a63-maas-api-service-ca\") pod \"8d91a9f7-a1aa-416f-b442-0565d0ec3a63\" (UID: \"8d91a9f7-a1aa-416f-b442-0565d0ec3a63\") " Jul 28 18:53:49.565669 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:49.565646 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d91a9f7-a1aa-416f-b442-0565d0ec3a63-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d91a9f7-a1aa-416f-b442-0565d0ec3a63" (UID: "8d91a9f7-a1aa-416f-b442-0565d0ec3a63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:53:49.666858 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:49.666818 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d91a9f7-a1aa-416f-b442-0565d0ec3a63-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:53:50.280420 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:50.280389 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" Jul 28 18:53:50.280420 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:50.280408 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872" event={"ID":"8d91a9f7-a1aa-416f-b442-0565d0ec3a63","Type":"ContainerDied","Data":"35999670d132117b17fa5c83152fbe2c4cf86a0dae7a4d85f190092a4abfe24c"} Jul 28 18:53:50.280644 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:50.280454 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35999670d132117b17fa5c83152fbe2c4cf86a0dae7a4d85f190092a4abfe24c" Jul 28 18:53:51.538360 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.538321 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k"] Jul 28 18:53:51.538806 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.538524 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" podUID="481a9d84-3a98-4398-895b-41ac5f683a11" containerName="maas-api" containerID="cri-o://30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599" gracePeriod=30 Jul 28 18:53:51.556555 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.556523 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:51.588843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.588795 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd"] Jul 28 18:53:51.589089 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.589061 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" podUID="3f8c4fc6-878e-4383-905d-ca0a56104f9c" containerName="payload-processing" containerID="cri-o://7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56" gracePeriod=30 Jul 28 18:53:51.661789 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.661731 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:51.789955 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.789815 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:51.828401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.828369 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872"] Jul 28 18:53:51.832064 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.832025 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-94ebbf-65872"] Jul 28 18:53:51.851943 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.851913 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:51.880468 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.880442 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:51.880711 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.880665 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:51.991394 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991354 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4stqn\" (UniqueName: \"kubernetes.io/projected/3f8c4fc6-878e-4383-905d-ca0a56104f9c-kube-api-access-4stqn\") pod \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " Jul 28 18:53:51.991610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991459 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzgs8\" (UniqueName: \"kubernetes.io/projected/481a9d84-3a98-4398-895b-41ac5f683a11-kube-api-access-hzgs8\") pod \"481a9d84-3a98-4398-895b-41ac5f683a11\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " Jul 28 18:53:51.991610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991512 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481a9d84-3a98-4398-895b-41ac5f683a11-openshift-service-ca\") pod \"481a9d84-3a98-4398-895b-41ac5f683a11\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " Jul 28 18:53:51.991610 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991557 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c4fc6-878e-4383-905d-ca0a56104f9c-plugins-config-volume\") pod \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\" (UID: \"3f8c4fc6-878e-4383-905d-ca0a56104f9c\") " Jul 28 18:53:51.991828 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991622 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481a9d84-3a98-4398-895b-41ac5f683a11-maas-api-tls\") pod \"481a9d84-3a98-4398-895b-41ac5f683a11\" (UID: \"481a9d84-3a98-4398-895b-41ac5f683a11\") " Jul 28 18:53:51.991983 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991952 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481a9d84-3a98-4398-895b-41ac5f683a11-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "481a9d84-3a98-4398-895b-41ac5f683a11" (UID: "481a9d84-3a98-4398-895b-41ac5f683a11"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:53:51.992061 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.991999 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f8c4fc6-878e-4383-905d-ca0a56104f9c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3f8c4fc6-878e-4383-905d-ca0a56104f9c" (UID: "3f8c4fc6-878e-4383-905d-ca0a56104f9c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:53:51.993735 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.993711 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8c4fc6-878e-4383-905d-ca0a56104f9c-kube-api-access-4stqn" (OuterVolumeSpecName: "kube-api-access-4stqn") pod "3f8c4fc6-878e-4383-905d-ca0a56104f9c" (UID: "3f8c4fc6-878e-4383-905d-ca0a56104f9c"). InnerVolumeSpecName "kube-api-access-4stqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:53:51.993856 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.993827 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481a9d84-3a98-4398-895b-41ac5f683a11-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "481a9d84-3a98-4398-895b-41ac5f683a11" (UID: "481a9d84-3a98-4398-895b-41ac5f683a11"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:53:51.993930 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:51.993898 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481a9d84-3a98-4398-895b-41ac5f683a11-kube-api-access-hzgs8" (OuterVolumeSpecName: "kube-api-access-hzgs8") pod "481a9d84-3a98-4398-895b-41ac5f683a11" (UID: "481a9d84-3a98-4398-895b-41ac5f683a11"). InnerVolumeSpecName "kube-api-access-hzgs8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:53:52.093112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.093013 2575 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f8c4fc6-878e-4383-905d-ca0a56104f9c-plugins-config-volume\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:53:52.093112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.093049 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/481a9d84-3a98-4398-895b-41ac5f683a11-maas-api-tls\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:53:52.093112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.093063 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4stqn\" (UniqueName: \"kubernetes.io/projected/3f8c4fc6-878e-4383-905d-ca0a56104f9c-kube-api-access-4stqn\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:53:52.093112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.093075 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzgs8\" (UniqueName: \"kubernetes.io/projected/481a9d84-3a98-4398-895b-41ac5f683a11-kube-api-access-hzgs8\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:53:52.093112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.093087 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/481a9d84-3a98-4398-895b-41ac5f683a11-openshift-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:53:52.206388 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.206353 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:52.294406 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.294373 2575 generic.go:356] "Generic (PLEG): container finished" podID="3f8c4fc6-878e-4383-905d-ca0a56104f9c" containerID="7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56" exitCode=0 Jul 28 18:53:52.294600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.294418 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" Jul 28 18:53:52.294600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.294477 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" event={"ID":"3f8c4fc6-878e-4383-905d-ca0a56104f9c","Type":"ContainerDied","Data":"7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56"} Jul 28 18:53:52.294600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.294529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd" event={"ID":"3f8c4fc6-878e-4383-905d-ca0a56104f9c","Type":"ContainerDied","Data":"bcd8cac77e8a4438c3735b9d0d26b06963373d281ae2a271aeffee5d53ad2f7a"} Jul 28 18:53:52.294600 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.294556 2575 scope.go:117] "RemoveContainer" containerID="7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56" Jul 28 18:53:52.296219 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.296189 2575 generic.go:356] "Generic (PLEG): container finished" podID="481a9d84-3a98-4398-895b-41ac5f683a11" containerID="30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599" exitCode=0 Jul 28 18:53:52.296356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.296252 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" Jul 28 18:53:52.296356 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.296260 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" event={"ID":"481a9d84-3a98-4398-895b-41ac5f683a11","Type":"ContainerDied","Data":"30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599"} Jul 28 18:53:52.296478 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.296389 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k" event={"ID":"481a9d84-3a98-4398-895b-41ac5f683a11","Type":"ContainerDied","Data":"7931858bc6854ea203d5dfb0f2106a8d16d03f3219980f3d006d77af4c864f08"} Jul 28 18:53:52.305838 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.305618 2575 scope.go:117] "RemoveContainer" containerID="7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56" Jul 28 18:53:52.306091 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:53:52.306068 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56\": container with ID starting with 7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56 not found: ID does not exist" containerID="7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56" Jul 28 18:53:52.306184 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.306099 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56"} err="failed to get container status \"7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56\": rpc error: code = NotFound desc = could not find container \"7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56\": container with ID starting with 7f5319d29f693fc6464e1c76f70e607abb93a964c435bcb008a53c5ea2244e56 not found: ID does not exist" Jul 28 18:53:52.306184 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.306117 2575 scope.go:117] "RemoveContainer" containerID="30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599" Jul 28 18:53:52.315146 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.315121 2575 scope.go:117] "RemoveContainer" containerID="30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599" Jul 28 18:53:52.315425 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:53:52.315406 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599\": container with ID starting with 30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599 not found: ID does not exist" containerID="30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599" Jul 28 18:53:52.315487 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.315434 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599"} err="failed to get container status \"30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599\": rpc error: code = NotFound desc = could not find container \"30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599\": container with ID starting with 30d87f318d177a085eb6846051a4c98b6cb2c9871f6b129bd085a562ecfb9599 not found: ID does not exist" Jul 28 18:53:52.321932 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.321902 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k"] Jul 28 18:53:52.325949 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.325920 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-94ebbf-5b99cf8898-2wv5k"] Jul 28 18:53:52.340383 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.340349 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd"] Jul 28 18:53:52.347842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:52.347761 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-94ebbf-5659ff8bd4-lxbnd"] Jul 28 18:53:53.060960 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:53.060922 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8c4fc6-878e-4383-905d-ca0a56104f9c" path="/var/lib/kubelet/pods/3f8c4fc6-878e-4383-905d-ca0a56104f9c/volumes" Jul 28 18:53:53.061338 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:53.061325 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481a9d84-3a98-4398-895b-41ac5f683a11" path="/var/lib/kubelet/pods/481a9d84-3a98-4398-895b-41ac5f683a11/volumes" Jul 28 18:53:53.061626 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:53.061614 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d91a9f7-a1aa-416f-b442-0565d0ec3a63" path="/var/lib/kubelet/pods/8d91a9f7-a1aa-416f-b442-0565d0ec3a63/volumes" Jul 28 18:53:58.442213 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:58.442182 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:58.563926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:58.563880 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:58.766375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:58.766278 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:53:59.036045 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:53:59.035947 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:02.556751 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.556707 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7"] Jul 28 18:54:02.557341 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557318 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481a9d84-3a98-4398-895b-41ac5f683a11" containerName="maas-api" Jul 28 18:54:02.557431 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557343 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="481a9d84-3a98-4398-895b-41ac5f683a11" containerName="maas-api" Jul 28 18:54:02.557431 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557357 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d91a9f7-a1aa-416f-b442-0565d0ec3a63" containerName="revoke-keys" Jul 28 18:54:02.557431 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557366 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d91a9f7-a1aa-416f-b442-0565d0ec3a63" containerName="revoke-keys" Jul 28 18:54:02.557431 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557382 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8c4fc6-878e-4383-905d-ca0a56104f9c" containerName="payload-processing" Jul 28 18:54:02.557431 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557392 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8c4fc6-878e-4383-905d-ca0a56104f9c" containerName="payload-processing" Jul 28 18:54:02.557717 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557526 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8c4fc6-878e-4383-905d-ca0a56104f9c" containerName="payload-processing" Jul 28 18:54:02.557717 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557544 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="481a9d84-3a98-4398-895b-41ac5f683a11" containerName="maas-api" Jul 28 18:54:02.557717 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.557557 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d91a9f7-a1aa-416f-b442-0565d0ec3a63" containerName="revoke-keys" Jul 28 18:54:02.560803 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.560775 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:02.563384 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.563365 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:54:02.568103 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.568073 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7"] Jul 28 18:54:02.588125 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.588086 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0139cdb2-5def-4c86-a497-fc26ed8289ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-473596-92ft7\" (UID: \"0139cdb2-5def-4c86-a497-fc26ed8289ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:02.689141 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.689091 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0139cdb2-5def-4c86-a497-fc26ed8289ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-473596-92ft7\" (UID: \"0139cdb2-5def-4c86-a497-fc26ed8289ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:02.689778 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.689757 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0139cdb2-5def-4c86-a497-fc26ed8289ab-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-473596-92ft7\" (UID: \"0139cdb2-5def-4c86-a497-fc26ed8289ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:02.873891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:02.873795 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:03.015877 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:03.015846 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7"] Jul 28 18:54:03.018034 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:54:03.017998 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0139cdb2_5def_4c86_a497_fc26ed8289ab.slice/crio-3c74d38dbe464b2922be0005d58df7bfae29e1ebfdcefa7509c6d4509211641e WatchSource:0}: Error finding container 3c74d38dbe464b2922be0005d58df7bfae29e1ebfdcefa7509c6d4509211641e: Status 404 returned error can't find the container with id 3c74d38dbe464b2922be0005d58df7bfae29e1ebfdcefa7509c6d4509211641e Jul 28 18:54:03.372434 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:03.372386 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" event={"ID":"0139cdb2-5def-4c86-a497-fc26ed8289ab","Type":"ContainerStarted","Data":"b551c8325542edde2245873250f68a995e0ba7f5beb341c58d12bd7cb73f73e6"} Jul 28 18:54:03.372626 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:03.372446 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" event={"ID":"0139cdb2-5def-4c86-a497-fc26ed8289ab","Type":"ContainerStarted","Data":"3c74d38dbe464b2922be0005d58df7bfae29e1ebfdcefa7509c6d4509211641e"} Jul 28 18:54:03.387071 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:03.387018 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" podStartSLOduration=1.3870037339999999 podStartE2EDuration="1.387003734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:54:03.386380602 +0000 UTC m=+3440.801037618" watchObservedRunningTime="2026-07-28 18:54:03.387003734 +0000 UTC m=+3440.801660733" Jul 28 18:54:04.379912 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:04.379874 2575 generic.go:356] "Generic (PLEG): container finished" podID="0139cdb2-5def-4c86-a497-fc26ed8289ab" containerID="b551c8325542edde2245873250f68a995e0ba7f5beb341c58d12bd7cb73f73e6" exitCode=0 Jul 28 18:54:04.380327 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:04.379965 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" event={"ID":"0139cdb2-5def-4c86-a497-fc26ed8289ab","Type":"ContainerDied","Data":"b551c8325542edde2245873250f68a995e0ba7f5beb341c58d12bd7cb73f73e6"} Jul 28 18:54:05.526038 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:05.526011 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:05.612977 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:05.612939 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0139cdb2-5def-4c86-a497-fc26ed8289ab-maas-api-service-ca\") pod \"0139cdb2-5def-4c86-a497-fc26ed8289ab\" (UID: \"0139cdb2-5def-4c86-a497-fc26ed8289ab\") " Jul 28 18:54:05.613336 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:05.613309 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0139cdb2-5def-4c86-a497-fc26ed8289ab-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0139cdb2-5def-4c86-a497-fc26ed8289ab" (UID: "0139cdb2-5def-4c86-a497-fc26ed8289ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:54:05.713967 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:05.713871 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0139cdb2-5def-4c86-a497-fc26ed8289ab-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:54:06.392752 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:06.392718 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" Jul 28 18:54:06.392752 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:06.392737 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7" event={"ID":"0139cdb2-5def-4c86-a497-fc26ed8289ab","Type":"ContainerDied","Data":"3c74d38dbe464b2922be0005d58df7bfae29e1ebfdcefa7509c6d4509211641e"} Jul 28 18:54:06.392978 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:06.392788 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c74d38dbe464b2922be0005d58df7bfae29e1ebfdcefa7509c6d4509211641e" Jul 28 18:54:11.504472 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:11.504429 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:11.604594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:11.604557 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:18.152241 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.152207 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:18.242804 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.242771 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:18.385102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.385068 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:18.429362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.429270 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7"] Jul 28 18:54:18.433317 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.433284 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-473596-92ft7"] Jul 28 18:54:18.446998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.446967 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:18.822745 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:18.822628 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:19.059299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:19.059262 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0139cdb2-5def-4c86-a497-fc26ed8289ab" path="/var/lib/kubelet/pods/0139cdb2-5def-4c86-a497-fc26ed8289ab/volumes" Jul 28 18:54:23.857201 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:23.857168 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:54:33.317847 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:54:33.317809 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:01.135710 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:01.135652 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:01.381471 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:01.381426 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:06.902375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:06.902280 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:06.971594 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:06.971561 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:07.162598 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:07.162491 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9"] Jul 28 18:55:07.162839 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:07.162794 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" containerID="cri-o://97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6" gracePeriod=30 Jul 28 18:55:07.183816 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:07.183786 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:07.413582 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:07.413473 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:12.425170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.425090 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:55:12.558519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558481 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-credential-socket\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.558519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558528 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istiod-ca-cert\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.558869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558549 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-token\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.558869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558570 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-envoy\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.558869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558757 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-socket\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.558869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558811 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-certs\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.558869 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558856 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-data\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.559138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558887 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-podinfo\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.559138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558914 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x29m\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-kube-api-access-7x29m\") pod \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\" (UID: \"4ea639b1-f46f-4ed6-8bef-f1dde540e607\") " Jul 28 18:55:12.559138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.558929 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:12.559138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559049 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:55:12.559138 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559062 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:12.559389 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559190 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:12.559447 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559364 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.559447 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559416 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.559447 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559432 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.559447 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.559446 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.560240 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.560206 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:12.560926 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.560892 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:12.561055 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.561039 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:55:12.561122 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.561073 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-kube-api-access-7x29m" (OuterVolumeSpecName: "kube-api-access-7x29m") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "kube-api-access-7x29m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:55:12.561178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.561125 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ea639b1-f46f-4ed6-8bef-f1dde540e607" (UID: "4ea639b1-f46f-4ed6-8bef-f1dde540e607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:55:12.660491 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.660444 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.660491 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.660480 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.660491 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.660492 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.660491 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.660503 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ea639b1-f46f-4ed6-8bef-f1dde540e607-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.660819 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.660515 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x29m\" (UniqueName: \"kubernetes.io/projected/4ea639b1-f46f-4ed6-8bef-f1dde540e607-kube-api-access-7x29m\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:12.807322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.807227 2575 generic.go:356] "Generic (PLEG): container finished" podID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerID="97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6" exitCode=0 Jul 28 18:55:12.807322 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.807276 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" Jul 28 18:55:12.807505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.807316 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" event={"ID":"4ea639b1-f46f-4ed6-8bef-f1dde540e607","Type":"ContainerDied","Data":"97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6"} Jul 28 18:55:12.807505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.807374 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9" event={"ID":"4ea639b1-f46f-4ed6-8bef-f1dde540e607","Type":"ContainerDied","Data":"e193f3c57d8d1e7f34b1ed09aa716c7f7178e71fc72dcb95c267cb70d44b470f"} Jul 28 18:55:12.807505 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.807397 2575 scope.go:117] "RemoveContainer" containerID="97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6" Jul 28 18:55:12.818061 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.818036 2575 scope.go:117] "RemoveContainer" containerID="734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6" Jul 28 18:55:12.827459 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.827437 2575 scope.go:117] "RemoveContainer" containerID="97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6" Jul 28 18:55:12.827832 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:55:12.827806 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6\": container with ID starting with 97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6 not found: ID does not exist" containerID="97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6" Jul 28 18:55:12.827945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.827838 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6"} err="failed to get container status \"97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6\": rpc error: code = NotFound desc = could not find container \"97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6\": container with ID starting with 97cbcd844ef9d96855d9153b0e4f7a825db9bfd61932c35180e1015ef5df9ca6 not found: ID does not exist" Jul 28 18:55:12.827945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.827857 2575 scope.go:117] "RemoveContainer" containerID="734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6" Jul 28 18:55:12.828164 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:55:12.828140 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6\": container with ID starting with 734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6 not found: ID does not exist" containerID="734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6" Jul 28 18:55:12.828228 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.828169 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6"} err="failed to get container status \"734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6\": rpc error: code = NotFound desc = could not find container \"734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6\": container with ID starting with 734000474dc53ad786690728438a961a70b0dcc4beec46b72b429d44419277d6 not found: ID does not exist" Jul 28 18:55:12.831432 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.831399 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9"] Jul 28 18:55:12.838590 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:12.838554 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-473596-openshift-default-7c65dffddb-df2d9"] Jul 28 18:55:13.060316 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:13.060226 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" path="/var/lib/kubelet/pods/4ea639b1-f46f-4ed6-8bef-f1dde540e607/volumes" Jul 28 18:55:18.010866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.010828 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn"] Jul 28 18:55:18.011335 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.011086 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" podUID="1181c218-a2b0-405b-9889-53ed1e913afc" containerName="maas-api" containerID="cri-o://7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4" gracePeriod=30 Jul 28 18:55:18.042353 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.042326 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:18.110998 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.110963 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:18.172878 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.172841 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:18.236936 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.236903 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:18.276778 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.276710 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:55:18.415619 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.415575 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls\") pod \"1181c218-a2b0-405b-9889-53ed1e913afc\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " Jul 28 18:55:18.415837 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.415679 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xf6vq\" (UniqueName: \"kubernetes.io/projected/1181c218-a2b0-405b-9889-53ed1e913afc-kube-api-access-xf6vq\") pod \"1181c218-a2b0-405b-9889-53ed1e913afc\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " Jul 28 18:55:18.415837 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.415802 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1181c218-a2b0-405b-9889-53ed1e913afc-openshift-service-ca\") pod \"1181c218-a2b0-405b-9889-53ed1e913afc\" (UID: \"1181c218-a2b0-405b-9889-53ed1e913afc\") " Jul 28 18:55:18.416154 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.416130 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1181c218-a2b0-405b-9889-53ed1e913afc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1181c218-a2b0-405b-9889-53ed1e913afc" (UID: "1181c218-a2b0-405b-9889-53ed1e913afc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:55:18.417842 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.417820 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1181c218-a2b0-405b-9889-53ed1e913afc" (UID: "1181c218-a2b0-405b-9889-53ed1e913afc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 18:55:18.417940 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.417861 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1181c218-a2b0-405b-9889-53ed1e913afc-kube-api-access-xf6vq" (OuterVolumeSpecName: "kube-api-access-xf6vq") pod "1181c218-a2b0-405b-9889-53ed1e913afc" (UID: "1181c218-a2b0-405b-9889-53ed1e913afc"). InnerVolumeSpecName "kube-api-access-xf6vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:55:18.516841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.516801 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf6vq\" (UniqueName: \"kubernetes.io/projected/1181c218-a2b0-405b-9889-53ed1e913afc-kube-api-access-xf6vq\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:18.516841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.516833 2575 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1181c218-a2b0-405b-9889-53ed1e913afc-openshift-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:18.516841 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.516845 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1181c218-a2b0-405b-9889-53ed1e913afc-maas-api-tls\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:18.597925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.597831 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:18.846391 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.846354 2575 generic.go:356] "Generic (PLEG): container finished" podID="1181c218-a2b0-405b-9889-53ed1e913afc" containerID="7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4" exitCode=0 Jul 28 18:55:18.846570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.846403 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" Jul 28 18:55:18.846570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.846399 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" event={"ID":"1181c218-a2b0-405b-9889-53ed1e913afc","Type":"ContainerDied","Data":"7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4"} Jul 28 18:55:18.846570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.846529 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn" event={"ID":"1181c218-a2b0-405b-9889-53ed1e913afc","Type":"ContainerDied","Data":"b7a3ceae45d84dfa97cb67ad4129ad94ecaf3288a2735c2201563f5a41e2a79b"} Jul 28 18:55:18.846570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.846557 2575 scope.go:117] "RemoveContainer" containerID="7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4" Jul 28 18:55:18.856800 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.856770 2575 scope.go:117] "RemoveContainer" containerID="7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4" Jul 28 18:55:18.857382 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:55:18.857358 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4\": container with ID starting with 7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4 not found: ID does not exist" containerID="7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4" Jul 28 18:55:18.857457 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.857394 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4"} err="failed to get container status \"7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4\": rpc error: code = NotFound desc = could not find container \"7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4\": container with ID starting with 7d07a56dffc88a51440b523f28fd22af93e22c7a2a4a1502358fb71b2bf6aaf4 not found: ID does not exist" Jul 28 18:55:18.871139 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.871096 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn"] Jul 28 18:55:18.875714 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:18.875673 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5b063e-6dcb995667-xwqtn"] Jul 28 18:55:19.059890 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:19.059857 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1181c218-a2b0-405b-9889-53ed1e913afc" path="/var/lib/kubelet/pods/1181c218-a2b0-405b-9889-53ed1e913afc/volumes" Jul 28 18:55:24.713328 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:24.713288 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:24.814436 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:24.814398 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:25.004949 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.004864 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:25.239835 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.239799 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:25.374506 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374407 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg"] Jul 28 18:55:25.374895 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374880 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374897 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374912 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374918 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374926 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1181c218-a2b0-405b-9889-53ed1e913afc" containerName="maas-api" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374932 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="1181c218-a2b0-405b-9889-53ed1e913afc" containerName="maas-api" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374939 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0139cdb2-5def-4c86-a497-fc26ed8289ab" containerName="revoke-keys" Jul 28 18:55:25.374953 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.374945 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="0139cdb2-5def-4c86-a497-fc26ed8289ab" containerName="revoke-keys" Jul 28 18:55:25.375173 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.375016 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" Jul 28 18:55:25.375173 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.375025 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="0139cdb2-5def-4c86-a497-fc26ed8289ab" containerName="revoke-keys" Jul 28 18:55:25.375173 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.375035 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="1181c218-a2b0-405b-9889-53ed1e913afc" containerName="maas-api" Jul 28 18:55:25.379437 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.379414 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:25.382171 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.382138 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:55:25.383570 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.383529 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:55:25.385934 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.385907 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg"] Jul 28 18:55:25.477377 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.477327 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48619bab-4008-4577-b607-e858f80b42de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg\" (UID: \"48619bab-4008-4577-b607-e858f80b42de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:25.577923 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.577877 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48619bab-4008-4577-b607-e858f80b42de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg\" (UID: \"48619bab-4008-4577-b607-e858f80b42de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:25.578626 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.578598 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48619bab-4008-4577-b607-e858f80b42de-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg\" (UID: \"48619bab-4008-4577-b607-e858f80b42de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:25.693042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.693004 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:25.837145 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.837113 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg"] Jul 28 18:55:25.837978 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:55:25.837951 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48619bab_4008_4577_b607_e858f80b42de.slice/crio-8df17c3a7dbdb8456d41aae8f06d4152dc51734933111190a60f4029783d97e2 WatchSource:0}: Error finding container 8df17c3a7dbdb8456d41aae8f06d4152dc51734933111190a60f4029783d97e2: Status 404 returned error can't find the container with id 8df17c3a7dbdb8456d41aae8f06d4152dc51734933111190a60f4029783d97e2 Jul 28 18:55:25.891837 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:25.891800 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" event={"ID":"48619bab-4008-4577-b607-e858f80b42de","Type":"ContainerStarted","Data":"8df17c3a7dbdb8456d41aae8f06d4152dc51734933111190a60f4029783d97e2"} Jul 28 18:55:26.101858 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:26.101824 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:26.899536 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:26.899493 2575 generic.go:356] "Generic (PLEG): container finished" podID="48619bab-4008-4577-b607-e858f80b42de" containerID="74ade2e50be67f2b634d9cb8b6bb16775c2acb053eb59bf3b15b1ef51fa7c5ff" exitCode=0 Jul 28 18:55:26.899994 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:26.899583 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" event={"ID":"48619bab-4008-4577-b607-e858f80b42de","Type":"ContainerDied","Data":"74ade2e50be67f2b634d9cb8b6bb16775c2acb053eb59bf3b15b1ef51fa7c5ff"} Jul 28 18:55:28.052462 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.052427 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:28.101375 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.101332 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48619bab-4008-4577-b607-e858f80b42de-maas-api-service-ca\") pod \"48619bab-4008-4577-b607-e858f80b42de\" (UID: \"48619bab-4008-4577-b607-e858f80b42de\") " Jul 28 18:55:28.101768 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.101725 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48619bab-4008-4577-b607-e858f80b42de-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48619bab-4008-4577-b607-e858f80b42de" (UID: "48619bab-4008-4577-b607-e858f80b42de"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:55:28.203114 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.203004 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48619bab-4008-4577-b607-e858f80b42de-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:28.913813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.913772 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" event={"ID":"48619bab-4008-4577-b607-e858f80b42de","Type":"ContainerDied","Data":"8df17c3a7dbdb8456d41aae8f06d4152dc51734933111190a60f4029783d97e2"} Jul 28 18:55:28.913813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.913803 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg" Jul 28 18:55:28.913995 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:28.913829 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8df17c3a7dbdb8456d41aae8f06d4152dc51734933111190a60f4029783d97e2" Jul 28 18:55:29.417537 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:29.417502 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:35.857154 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:35.857102 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:35.857940 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:35.857915 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr"] Jul 28 18:55:35.862888 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:35.862858 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5523b7-29754405-6lwrr"] Jul 28 18:55:35.956709 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:35.956655 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:36.070787 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:36.070751 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:36.105555 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:36.105505 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg"] Jul 28 18:55:36.108158 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:36.108093 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-5523b7-v67kg"] Jul 28 18:55:36.144944 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:36.144914 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:36.481777 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:36.481743 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:37.060603 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:37.060563 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48619bab-4008-4577-b607-e858f80b42de" path="/var/lib/kubelet/pods/48619bab-4008-4577-b607-e858f80b42de/volumes" Jul 28 18:55:37.061052 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:37.060947 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="499d13f7-b9db-45f4-912e-10fcd98eb579" path="/var/lib/kubelet/pods/499d13f7-b9db-45f4-912e-10fcd98eb579/volumes" Jul 28 18:55:42.554917 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:42.554881 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:42.629111 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:42.629070 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:42.791866 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:42.791826 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb"] Jul 28 18:55:42.792158 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:42.792042 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" containerID="cri-o://87db87915c843c42ce39dd4b69b652ce7bbc0575159dd2ba4e8e853b6f962edb" gracePeriod=30 Jul 28 18:55:42.812289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:42.812193 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:43.064571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.064484 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:43.188817 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.188774 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65"] Jul 28 18:55:43.189198 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.189183 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48619bab-4008-4577-b607-e858f80b42de" containerName="revoke-keys" Jul 28 18:55:43.189244 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.189199 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="48619bab-4008-4577-b607-e858f80b42de" containerName="revoke-keys" Jul 28 18:55:43.189280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.189264 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ea639b1-f46f-4ed6-8bef-f1dde540e607" containerName="istio-proxy" Jul 28 18:55:43.189280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.189276 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="48619bab-4008-4577-b607-e858f80b42de" containerName="revoke-keys" Jul 28 18:55:43.192773 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.192750 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:43.195289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.195254 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 28 18:55:43.195289 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.195255 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-xtqk5\"" Jul 28 18:55:43.202169 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.202126 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65"] Jul 28 18:55:43.252068 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.252023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e29dc21-eab4-49e0-a086-b3a444e22285-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65\" (UID: \"7e29dc21-eab4-49e0-a086-b3a444e22285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:43.353239 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.353129 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e29dc21-eab4-49e0-a086-b3a444e22285-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65\" (UID: \"7e29dc21-eab4-49e0-a086-b3a444e22285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:43.353834 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.353812 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e29dc21-eab4-49e0-a086-b3a444e22285-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65\" (UID: \"7e29dc21-eab4-49e0-a086-b3a444e22285\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:43.505310 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.505272 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:43.654042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.654008 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65"] Jul 28 18:55:43.655584 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:55:43.655555 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e29dc21_eab4_49e0_a086_b3a444e22285.slice/crio-0c238bb8d778dfb46b1cbb828497623f83bb594d01c50d8a74083915a1106bb8 WatchSource:0}: Error finding container 0c238bb8d778dfb46b1cbb828497623f83bb594d01c50d8a74083915a1106bb8: Status 404 returned error can't find the container with id 0c238bb8d778dfb46b1cbb828497623f83bb594d01c50d8a74083915a1106bb8 Jul 28 18:55:43.788112 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:43.788074 2575 scope.go:117] "RemoveContainer" containerID="0b4bb3cdb03ca8323f570090cb40e92dc3c5e5830a13817df35bd0044c49cdfa" Jul 28 18:55:44.013158 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:44.013109 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" event={"ID":"7e29dc21-eab4-49e0-a086-b3a444e22285","Type":"ContainerStarted","Data":"1019295618b9148dd1ff43c5c14c0e199d6033b25e013e51d16839b184e82067"} Jul 28 18:55:44.013405 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:44.013181 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" event={"ID":"7e29dc21-eab4-49e0-a086-b3a444e22285","Type":"ContainerStarted","Data":"0c238bb8d778dfb46b1cbb828497623f83bb594d01c50d8a74083915a1106bb8"} Jul 28 18:55:44.032786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:44.032727 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" podStartSLOduration=1.032707978 podStartE2EDuration="1.032707978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:55:44.030358201 +0000 UTC m=+3541.445015202" watchObservedRunningTime="2026-07-28 18:55:44.032707978 +0000 UTC m=+3541.447364977" Jul 28 18:55:45.027776 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:45.027729 2575 generic.go:356] "Generic (PLEG): container finished" podID="7e29dc21-eab4-49e0-a086-b3a444e22285" containerID="1019295618b9148dd1ff43c5c14c0e199d6033b25e013e51d16839b184e82067" exitCode=0 Jul 28 18:55:45.028214 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:45.027804 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" event={"ID":"7e29dc21-eab4-49e0-a086-b3a444e22285","Type":"ContainerDied","Data":"1019295618b9148dd1ff43c5c14c0e199d6033b25e013e51d16839b184e82067"} Jul 28 18:55:46.166000 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:46.165968 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:46.278573 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:46.278527 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e29dc21-eab4-49e0-a086-b3a444e22285-maas-api-service-ca\") pod \"7e29dc21-eab4-49e0-a086-b3a444e22285\" (UID: \"7e29dc21-eab4-49e0-a086-b3a444e22285\") " Jul 28 18:55:46.278898 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:46.278874 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e29dc21-eab4-49e0-a086-b3a444e22285-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e29dc21-eab4-49e0-a086-b3a444e22285" (UID: "7e29dc21-eab4-49e0-a086-b3a444e22285"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:55:46.379617 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:46.379580 2575 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e29dc21-eab4-49e0-a086-b3a444e22285-maas-api-service-ca\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:47.040113 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:47.040073 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" event={"ID":"7e29dc21-eab4-49e0-a086-b3a444e22285","Type":"ContainerDied","Data":"0c238bb8d778dfb46b1cbb828497623f83bb594d01c50d8a74083915a1106bb8"} Jul 28 18:55:47.040268 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:47.040128 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c238bb8d778dfb46b1cbb828497623f83bb594d01c50d8a74083915a1106bb8" Jul 28 18:55:47.040268 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:47.040089 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65" Jul 28 18:55:48.048914 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.048883 2575 generic.go:356] "Generic (PLEG): container finished" podID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerID="87db87915c843c42ce39dd4b69b652ce7bbc0575159dd2ba4e8e853b6f962edb" exitCode=0 Jul 28 18:55:48.049304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.048951 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" event={"ID":"9a7bec57-4ab1-49eb-9279-0faaaed504b9","Type":"ContainerDied","Data":"87db87915c843c42ce39dd4b69b652ce7bbc0575159dd2ba4e8e853b6f962edb"} Jul 28 18:55:48.049304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.049008 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" event={"ID":"9a7bec57-4ab1-49eb-9279-0faaaed504b9","Type":"ContainerDied","Data":"06383bec4bad582ee3e66e3a39a0735aa4bbbf1f21446fbdee6b2a526d8e0ba7"} Jul 28 18:55:48.049304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.049026 2575 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06383bec4bad582ee3e66e3a39a0735aa4bbbf1f21446fbdee6b2a526d8e0ba7" Jul 28 18:55:48.049304 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.049042 2575 scope.go:117] "RemoveContainer" containerID="6112a5150ad990af759b5043df9ca86622283b1535e59db5cf7b7a9ed58a4bdb" Jul 28 18:55:48.051183 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.051156 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:55:48.196843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196799 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-podinfo\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.196843 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196845 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istiod-ca-cert\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196864 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-data\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196888 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-token\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196929 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-certs\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196955 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-credential-socket\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.196991 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-socket\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197102 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197034 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-envoy\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197116 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5lz7j\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-kube-api-access-5lz7j\") pod \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\" (UID: \"9a7bec57-4ab1-49eb-9279-0faaaed504b9\") " Jul 28 18:55:48.197416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197307 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:48.197416 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197228 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:55:48.197559 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197502 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:48.197559 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197509 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.197559 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197531 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.197743 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197589 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:48.197925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.197889 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:48.199297 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.199264 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:55:48.199401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.199297 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:55:48.199401 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.199320 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:55:48.199513 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.199497 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-kube-api-access-5lz7j" (OuterVolumeSpecName: "kube-api-access-5lz7j") pod "9a7bec57-4ab1-49eb-9279-0faaaed504b9" (UID: "9a7bec57-4ab1-49eb-9279-0faaaed504b9"). InnerVolumeSpecName "kube-api-access-5lz7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:55:48.299010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.298965 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.299010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.299001 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lz7j\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-kube-api-access-5lz7j\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.299010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.299011 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.299010 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.299021 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.299290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.299030 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a7bec57-4ab1-49eb-9279-0faaaed504b9-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.299290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.299037 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:48.299290 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:48.299045 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a7bec57-4ab1-49eb-9279-0faaaed504b9-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:55:49.056613 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:49.056583 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb" Jul 28 18:55:49.092554 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:49.092519 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb"] Jul 28 18:55:49.098603 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:49.098569 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-5523b7-openshift-default-6df6675456-s5jpb"] Jul 28 18:55:51.059249 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:51.059214 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" path="/var/lib/kubelet/pods/9a7bec57-4ab1-49eb-9279-0faaaed504b9/volumes" Jul 28 18:55:53.608324 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.608279 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:53.610120 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.610093 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9"] Jul 28 18:55:53.612386 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.612363 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-10fb8f-29754405-vkkx9"] Jul 28 18:55:53.677723 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.677667 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:53.739044 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.739013 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:53.766450 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.766408 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65"] Jul 28 18:55:53.769468 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.769436 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-10fb8f-5fc65"] Jul 28 18:55:53.804046 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:53.804004 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:54.175233 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:54.175186 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:55:55.059134 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:55.059101 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e29dc21-eab4-49e0-a086-b3a444e22285" path="/var/lib/kubelet/pods/7e29dc21-eab4-49e0-a086-b3a444e22285/volumes" Jul 28 18:55:55.059520 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:55:55.059382 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa95f59-dca6-4fa4-90de-6fac445426e5" path="/var/lib/kubelet/pods/cfa95f59-dca6-4fa4-90de-6fac445426e5/volumes" Jul 28 18:56:00.114346 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:00.114310 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:56:00.373438 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:00.373335 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5"] Jul 28 18:56:00.373631 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:00.373607 2575 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" containerID="cri-o://556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445" gracePeriod=30 Jul 28 18:56:01.063374 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:01.063345 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57cf8ff8f9-nl86t_bf54ccf4-688f-4d82-9fe3-1fd5996367cc/authorino/0.log" Jul 28 18:56:01.464182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:01.464146 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:56:05.640625 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.640534 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:56:05.756997 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.756958 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-token\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757025 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-podinfo\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757052 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-envoy\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757124 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istiod-ca-cert\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757151 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2s7th\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-kube-api-access-2s7th\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757178 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757170 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-data\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757191 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-credential-socket\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757237 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-socket\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757473 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757277 2575 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-certs\") pod \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\" (UID: \"8d1b9351-0815-4bd8-8b06-b8933243e5fb\") " Jul 28 18:56:05.757654 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757622 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 18:56:05.757746 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757678 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:05.757814 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757623 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:05.757891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.757871 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:05.758173 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.758139 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:05.759406 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.759377 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 28 18:56:05.759532 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.759378 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-kube-api-access-2s7th" (OuterVolumeSpecName: "kube-api-access-2s7th") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "kube-api-access-2s7th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:56:05.759532 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.759447 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 18:56:05.759532 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.759497 2575 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8d1b9351-0815-4bd8-8b06-b8933243e5fb" (UID: "8d1b9351-0815-4bd8-8b06-b8933243e5fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 18:56:05.858062 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858018 2575 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-certs\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858062 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858055 2575 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-token\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858062 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858068 2575 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-podinfo\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858345 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858079 2575 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-envoy\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858345 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858088 2575 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istiod-ca-cert\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858345 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858096 2575 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s7th\" (UniqueName: \"kubernetes.io/projected/8d1b9351-0815-4bd8-8b06-b8933243e5fb-kube-api-access-2s7th\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858345 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858105 2575 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-istio-data\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858345 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858117 2575 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-credential-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:05.858345 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:05.858126 2575 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1b9351-0815-4bd8-8b06-b8933243e5fb-workload-socket\") on node \"ip-10-0-135-147.ec2.internal\" DevicePath \"\"" Jul 28 18:56:06.160182 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.160147 2575 generic.go:356] "Generic (PLEG): container finished" podID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerID="556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445" exitCode=0 Jul 28 18:56:06.160349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.160202 2575 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" Jul 28 18:56:06.160349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.160237 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" event={"ID":"8d1b9351-0815-4bd8-8b06-b8933243e5fb","Type":"ContainerDied","Data":"556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445"} Jul 28 18:56:06.160349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.160305 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5" event={"ID":"8d1b9351-0815-4bd8-8b06-b8933243e5fb","Type":"ContainerDied","Data":"393d2ac6705ac734064c38304ffcfc23a19f3ff0b014978b7151645818c65dbd"} Jul 28 18:56:06.160349 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.160337 2575 scope.go:117] "RemoveContainer" containerID="556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445" Jul 28 18:56:06.170991 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.170963 2575 scope.go:117] "RemoveContainer" containerID="76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b" Jul 28 18:56:06.181106 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.181078 2575 scope.go:117] "RemoveContainer" containerID="556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445" Jul 28 18:56:06.181749 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:56:06.181653 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445\": container with ID starting with 556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445 not found: ID does not exist" containerID="556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445" Jul 28 18:56:06.181886 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.181765 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445"} err="failed to get container status \"556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445\": rpc error: code = NotFound desc = could not find container \"556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445\": container with ID starting with 556d8a3c7c7981fba8516bd327fa260dc3936a9bd2d243e35da6170845b85445 not found: ID does not exist" Jul 28 18:56:06.181886 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.181809 2575 scope.go:117] "RemoveContainer" containerID="76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b" Jul 28 18:56:06.182112 ip-10-0-135-147 kubenswrapper[2575]: E0728 18:56:06.182093 2575 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b\": container with ID starting with 76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b not found: ID does not exist" containerID="76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b" Jul 28 18:56:06.182156 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.182120 2575 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b"} err="failed to get container status \"76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b\": rpc error: code = NotFound desc = could not find container \"76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b\": container with ID starting with 76e5f882eac73c335a323e254a45762a78c0e71f206e850312d2dcae8dcf662b not found: ID does not exist" Jul 28 18:56:06.184327 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.184301 2575 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5"] Jul 28 18:56:06.188622 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:06.188590 2575 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-10fb8f-openshift-default-756b566676-bq6r5"] Jul 28 18:56:07.062675 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:07.062638 2575 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" path="/var/lib/kubelet/pods/8d1b9351-0815-4bd8-8b06-b8933243e5fb/volumes" Jul 28 18:56:07.333231 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:07.333152 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fb95c4cb9-kmp44_f65179d2-f023-42c2-8f23-94791e9afc77/copy-manifests/0.log" Jul 28 18:56:07.340166 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:07.340141 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fb95c4cb9-kmp44_f65179d2-f023-42c2-8f23-94791e9afc77/manager/0.log" Jul 28 18:56:07.957919 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:07.957890 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754375-pxr4j_2d25e8a6-cbab-471f-b614-3470df76a572/cleanup/0.log" Jul 28 18:56:08.087891 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:08.087857 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754390-47xd8_dacfaa5a-65a1-489e-a545-e40eebaab5bb/cleanup/0.log" Jul 28 18:56:08.202945 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:08.202915 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29754405-gjxbq_b1980385-8dcb-450d-a8f8-71cf228b24b6/cleanup/0.log" Jul 28 18:56:09.144095 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.144065 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j_d183ac02-7138-4276-aeed-612fbd2e7b77/extract/0.log" Jul 28 18:56:09.150867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.150842 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j_d183ac02-7138-4276-aeed-612fbd2e7b77/util/0.log" Jul 28 18:56:09.158339 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.158315 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j_d183ac02-7138-4276-aeed-612fbd2e7b77/pull/0.log" Jul 28 18:56:09.274619 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.274591 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x_f403b2e1-5fcd-4a34-a161-5ad0795c1c71/util/0.log" Jul 28 18:56:09.282898 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.282864 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x_f403b2e1-5fcd-4a34-a161-5ad0795c1c71/pull/0.log" Jul 28 18:56:09.291080 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.291048 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x_f403b2e1-5fcd-4a34-a161-5ad0795c1c71/extract/0.log" Jul 28 18:56:09.407774 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.407672 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx_c7ac4b6b-4c33-4715-bf86-5ed24604d61b/util/0.log" Jul 28 18:56:09.415048 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.415024 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx_c7ac4b6b-4c33-4715-bf86-5ed24604d61b/pull/0.log" Jul 28 18:56:09.421919 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.421886 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx_c7ac4b6b-4c33-4715-bf86-5ed24604d61b/extract/0.log" Jul 28 18:56:09.545212 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.545181 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn_a9dc5f58-4d70-4e61-8d3c-55dec04a3e63/util/0.log" Jul 28 18:56:09.554933 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.554904 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn_a9dc5f58-4d70-4e61-8d3c-55dec04a3e63/pull/0.log" Jul 28 18:56:09.562090 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.562059 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn_a9dc5f58-4d70-4e61-8d3c-55dec04a3e63/extract/0.log" Jul 28 18:56:09.681495 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:09.681406 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57cf8ff8f9-nl86t_bf54ccf4-688f-4d82-9fe3-1fd5996367cc/authorino/0.log" Jul 28 18:56:10.511867 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:10.511829 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-d2qct_d9bec5f7-9577-4d87-b050-acd9b6b87490/manager/0.log" Jul 28 18:56:11.128031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:11.127997 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfd4f6bf9-cxdlb_d71a188f-e9d4-490c-902c-e25ef6dfb5b5/kube-auth-proxy/0.log" Jul 28 18:56:11.619146 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:11.619060 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tprs8_43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c/payload-processing/0.log" Jul 28 18:56:18.037274 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037233 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xtqp2/must-gather-sbnld"] Jul 28 18:56:18.037852 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037831 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" Jul 28 18:56:18.037905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037855 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" Jul 28 18:56:18.037905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037871 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" Jul 28 18:56:18.037905 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037880 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" Jul 28 18:56:18.038004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037910 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" Jul 28 18:56:18.038004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037920 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" Jul 28 18:56:18.038004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037939 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e29dc21-eab4-49e0-a086-b3a444e22285" containerName="revoke-keys" Jul 28 18:56:18.038004 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.037946 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e29dc21-eab4-49e0-a086-b3a444e22285" containerName="revoke-keys" Jul 28 18:56:18.038127 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038030 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" Jul 28 18:56:18.038127 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038041 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e29dc21-eab4-49e0-a086-b3a444e22285" containerName="revoke-keys" Jul 28 18:56:18.038127 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038054 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" Jul 28 18:56:18.038127 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038063 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" Jul 28 18:56:18.038248 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038140 2575 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" Jul 28 18:56:18.038248 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038149 2575 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a7bec57-4ab1-49eb-9279-0faaaed504b9" containerName="istio-proxy" Jul 28 18:56:18.038327 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.038253 2575 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d1b9351-0815-4bd8-8b06-b8933243e5fb" containerName="istio-proxy" Jul 28 18:56:18.043430 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.043402 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.046699 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.046653 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtqp2\"/\"kube-root-ca.crt\"" Jul 28 18:56:18.046855 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.046653 2575 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xtqp2\"/\"openshift-service-ca.crt\"" Jul 28 18:56:18.051974 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.051946 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtqp2/must-gather-sbnld"] Jul 28 18:56:18.067571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.067527 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kgtn\" (UniqueName: \"kubernetes.io/projected/86cd4b90-ee68-4120-ba83-d66bf34b373c-kube-api-access-4kgtn\") pod \"must-gather-sbnld\" (UID: \"86cd4b90-ee68-4120-ba83-d66bf34b373c\") " pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.067571 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.067582 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86cd4b90-ee68-4120-ba83-d66bf34b373c-must-gather-output\") pod \"must-gather-sbnld\" (UID: \"86cd4b90-ee68-4120-ba83-d66bf34b373c\") " pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.068398 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.068374 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:56:18.165575 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.165528 2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" limit=1024 Jul 28 18:56:18.168670 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.168636 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86cd4b90-ee68-4120-ba83-d66bf34b373c-must-gather-output\") pod \"must-gather-sbnld\" (UID: \"86cd4b90-ee68-4120-ba83-d66bf34b373c\") " pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.168836 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.168801 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kgtn\" (UniqueName: \"kubernetes.io/projected/86cd4b90-ee68-4120-ba83-d66bf34b373c-kube-api-access-4kgtn\") pod \"must-gather-sbnld\" (UID: \"86cd4b90-ee68-4120-ba83-d66bf34b373c\") " pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.168990 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.168971 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86cd4b90-ee68-4120-ba83-d66bf34b373c-must-gather-output\") pod \"must-gather-sbnld\" (UID: \"86cd4b90-ee68-4120-ba83-d66bf34b373c\") " pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.178326 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.178298 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kgtn\" (UniqueName: \"kubernetes.io/projected/86cd4b90-ee68-4120-ba83-d66bf34b373c-kube-api-access-4kgtn\") pod \"must-gather-sbnld\" (UID: \"86cd4b90-ee68-4120-ba83-d66bf34b373c\") " pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.354158 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.354054 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtqp2/must-gather-sbnld" Jul 28 18:56:18.501209 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:18.501179 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtqp2/must-gather-sbnld"] Jul 28 18:56:18.503235 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:56:18.503203 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86cd4b90_ee68_4120_ba83_d66bf34b373c.slice/crio-ea878892406a4171ce0f4ef557b5918ccd7bfcd53dc3906ff7923164091cb80e WatchSource:0}: Error finding container ea878892406a4171ce0f4ef557b5918ccd7bfcd53dc3906ff7923164091cb80e: Status 404 returned error can't find the container with id ea878892406a4171ce0f4ef557b5918ccd7bfcd53dc3906ff7923164091cb80e Jul 28 18:56:19.242635 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:19.242590 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtqp2/must-gather-sbnld" event={"ID":"86cd4b90-ee68-4120-ba83-d66bf34b373c","Type":"ContainerStarted","Data":"41a7160999fb50b30b505e37b06408c3c6ca599671a776b5ebb2e1fdad6ca100"} Jul 28 18:56:19.243042 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:19.242655 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtqp2/must-gather-sbnld" event={"ID":"86cd4b90-ee68-4120-ba83-d66bf34b373c","Type":"ContainerStarted","Data":"ea878892406a4171ce0f4ef557b5918ccd7bfcd53dc3906ff7923164091cb80e"} Jul 28 18:56:20.259771 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:20.259721 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtqp2/must-gather-sbnld" event={"ID":"86cd4b90-ee68-4120-ba83-d66bf34b373c","Type":"ContainerStarted","Data":"01a342559770c94ceb6eef697d87d9eed340ec1127d9b4f5086fa55bdda18d4a"} Jul 28 18:56:20.279015 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:20.278953 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xtqp2/must-gather-sbnld" podStartSLOduration=1.645120259 podStartE2EDuration="2.27893356s" totalImagesPullingTime="633.813301ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 18:56:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:20.276478697 +0000 UTC m=+3577.691135700" watchObservedRunningTime="2026-07-28 18:56:20.27893356 +0000 UTC m=+3577.693590562" Jul 28 18:56:20.989678 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:20.989609 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-ztp68_63e15748-3635-4ac4-be53-09c308ba90bb/global-pull-secret-syncer/0.log" Jul 28 18:56:21.044479 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:21.044442 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6gtgh_df10c201-bf85-4f6c-b7b1-e79bc719405c/konnectivity-agent/0.log" Jul 28 18:56:21.230226 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:21.230190 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-147.ec2.internal_545c1a00c368df77d9e72312b6e155db/haproxy/0.log" Jul 28 18:56:26.365826 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.365793 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j_d183ac02-7138-4276-aeed-612fbd2e7b77/extract/0.log" Jul 28 18:56:26.392086 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.392056 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j_d183ac02-7138-4276-aeed-612fbd2e7b77/util/0.log" Jul 28 18:56:26.434240 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.434176 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759n546j_d183ac02-7138-4276-aeed-612fbd2e7b77/pull/0.log" Jul 28 18:56:26.469031 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.469002 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x_f403b2e1-5fcd-4a34-a161-5ad0795c1c71/extract/0.log" Jul 28 18:56:26.498007 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.497972 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x_f403b2e1-5fcd-4a34-a161-5ad0795c1c71/util/0.log" Jul 28 18:56:26.527544 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.527510 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0nsx9x_f403b2e1-5fcd-4a34-a161-5ad0795c1c71/pull/0.log" Jul 28 18:56:26.561070 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.561032 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx_c7ac4b6b-4c33-4715-bf86-5ed24604d61b/extract/0.log" Jul 28 18:56:26.588778 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.588739 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx_c7ac4b6b-4c33-4715-bf86-5ed24604d61b/util/0.log" Jul 28 18:56:26.616256 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.616162 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73j9scx_c7ac4b6b-4c33-4715-bf86-5ed24604d61b/pull/0.log" Jul 28 18:56:26.649169 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.649134 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn_a9dc5f58-4d70-4e61-8d3c-55dec04a3e63/extract/0.log" Jul 28 18:56:26.676517 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.676450 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn_a9dc5f58-4d70-4e61-8d3c-55dec04a3e63/util/0.log" Jul 28 18:56:26.704565 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:26.704531 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1bslrn_a9dc5f58-4d70-4e61-8d3c-55dec04a3e63/pull/0.log" Jul 28 18:56:27.169749 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:27.169710 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-57cf8ff8f9-nl86t_bf54ccf4-688f-4d82-9fe3-1fd5996367cc/authorino/0.log" Jul 28 18:56:27.856437 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:27.856401 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-d2qct_d9bec5f7-9577-4d87-b050-acd9b6b87490/manager/0.log" Jul 28 18:56:29.459657 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.459615 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/alertmanager/0.log" Jul 28 18:56:29.490372 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.490338 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/config-reloader/0.log" Jul 28 18:56:29.517629 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.517601 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/kube-rbac-proxy-web/0.log" Jul 28 18:56:29.546797 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.546767 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/kube-rbac-proxy/0.log" Jul 28 18:56:29.573561 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.573510 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/kube-rbac-proxy-metric/0.log" Jul 28 18:56:29.615577 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.615542 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/prom-label-proxy/0.log" Jul 28 18:56:29.637990 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:29.637948 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_59009595-f644-4b3b-8305-45a058f2799a/init-config-reloader/0.log" Jul 28 18:56:30.035188 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.035153 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbg9l_e677163f-dbdd-48f9-b7a2-4a3b799d7fad/node-exporter/0.log" Jul 28 18:56:30.062299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.062267 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbg9l_e677163f-dbdd-48f9-b7a2-4a3b799d7fad/kube-rbac-proxy/0.log" Jul 28 18:56:30.087561 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.087530 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mbg9l_e677163f-dbdd-48f9-b7a2-4a3b799d7fad/init-textfile/0.log" Jul 28 18:56:30.117280 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.117244 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-8chls_8bf3cdbc-00eb-48b7-81a1-936a4983dd1c/kube-rbac-proxy-main/0.log" Jul 28 18:56:30.144621 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.144589 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-8chls_8bf3cdbc-00eb-48b7-81a1-936a4983dd1c/kube-rbac-proxy-self/0.log" Jul 28 18:56:30.170362 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.170329 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-8chls_8bf3cdbc-00eb-48b7-81a1-936a4983dd1c/openshift-state-metrics/0.log" Jul 28 18:56:30.401225 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.401191 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kj4t7_ecd696fc-76ef-4036-9cad-322083da49f9/prometheus-operator/0.log" Jul 28 18:56:30.420301 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.420259 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-kj4t7_ecd696fc-76ef-4036-9cad-322083da49f9/kube-rbac-proxy/0.log" Jul 28 18:56:30.459591 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.459556 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-9k746_b61a3559-21e8-4b9f-b2f3-efe1ed805953/prometheus-operator-admission-webhook/0.log" Jul 28 18:56:30.495829 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.495798 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d784b7d86-t954g_9233ebf0-e875-4fc2-a66c-37d7eed0f203/telemeter-client/0.log" Jul 28 18:56:30.518514 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.518486 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d784b7d86-t954g_9233ebf0-e875-4fc2-a66c-37d7eed0f203/reload/0.log" Jul 28 18:56:30.540915 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.540885 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6d784b7d86-t954g_9233ebf0-e875-4fc2-a66c-37d7eed0f203/kube-rbac-proxy/0.log" Jul 28 18:56:30.573519 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.573493 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4cdbfd9b-pgv2v_74c783e0-b4db-4820-a21a-f81a829763bd/thanos-query/0.log" Jul 28 18:56:30.601925 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.601896 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4cdbfd9b-pgv2v_74c783e0-b4db-4820-a21a-f81a829763bd/kube-rbac-proxy-web/0.log" Jul 28 18:56:30.628420 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.628377 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4cdbfd9b-pgv2v_74c783e0-b4db-4820-a21a-f81a829763bd/kube-rbac-proxy/0.log" Jul 28 18:56:30.651784 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.651693 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4cdbfd9b-pgv2v_74c783e0-b4db-4820-a21a-f81a829763bd/prom-label-proxy/0.log" Jul 28 18:56:30.673133 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.673105 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4cdbfd9b-pgv2v_74c783e0-b4db-4820-a21a-f81a829763bd/kube-rbac-proxy-rules/0.log" Jul 28 18:56:30.696348 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:30.696319 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-c4cdbfd9b-pgv2v_74c783e0-b4db-4820-a21a-f81a829763bd/kube-rbac-proxy-metrics/0.log" Jul 28 18:56:31.402074 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.402036 2575 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv"] Jul 28 18:56:31.409273 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.408737 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.412197 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.412141 2575 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xtqp2\"/\"default-dockercfg-rdvgc\"" Jul 28 18:56:31.419233 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.419199 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv"] Jul 28 18:56:31.519299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.515023 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-proc\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.519299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.515100 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dls6s\" (UniqueName: \"kubernetes.io/projected/43ebf55e-61da-4713-93bb-62e97d1475bb-kube-api-access-dls6s\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.519299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.515186 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-sys\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.519299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.515355 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-podres\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.519299 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.515542 2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-lib-modules\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617050 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.616975 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dls6s\" (UniqueName: \"kubernetes.io/projected/43ebf55e-61da-4713-93bb-62e97d1475bb-kube-api-access-dls6s\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617266 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617104 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-sys\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617266 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617207 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-podres\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617266 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617236 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-lib-modules\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617266 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617252 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-sys\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617289 2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-proc\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617379 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-proc\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617419 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-podres\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.617502 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.617432 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/43ebf55e-61da-4713-93bb-62e97d1475bb-lib-modules\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.625475 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.625437 2575 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dls6s\" (UniqueName: \"kubernetes.io/projected/43ebf55e-61da-4713-93bb-62e97d1475bb-kube-api-access-dls6s\") pod \"perf-node-gather-daemonset-jsdtv\" (UID: \"43ebf55e-61da-4713-93bb-62e97d1475bb\") " pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.725611 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.725508 2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:31.921708 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:31.920665 2575 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv"] Jul 28 18:56:31.926770 ip-10-0-135-147 kubenswrapper[2575]: W0728 18:56:31.926726 2575 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod43ebf55e_61da_4713_93bb_62e97d1475bb.slice/crio-a09f164a83afa5d051eeee2f3a0f7af70b0627707caa4b1fa201ce544f3e1867 WatchSource:0}: Error finding container a09f164a83afa5d051eeee2f3a0f7af70b0627707caa4b1fa201ce544f3e1867: Status 404 returned error can't find the container with id a09f164a83afa5d051eeee2f3a0f7af70b0627707caa4b1fa201ce544f3e1867 Jul 28 18:56:32.378980 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:32.378937 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" event={"ID":"43ebf55e-61da-4713-93bb-62e97d1475bb","Type":"ContainerStarted","Data":"5eca375d9d892679562f96c77861bf286c64b6f62b160bf28adcb4c1a742144f"} Jul 28 18:56:32.379170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:32.379010 2575 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" event={"ID":"43ebf55e-61da-4713-93bb-62e97d1475bb","Type":"ContainerStarted","Data":"a09f164a83afa5d051eeee2f3a0f7af70b0627707caa4b1fa201ce544f3e1867"} Jul 28 18:56:32.379170 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:32.379042 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:32.396890 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:32.396835 2575 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" podStartSLOduration=1.396819363 podStartE2EDuration="1.396819363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 18:56:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 18:56:32.395213408 +0000 UTC m=+3589.809870410" watchObservedRunningTime="2026-07-28 18:56:32.396819363 +0000 UTC m=+3589.811476364" Jul 28 18:56:32.928907 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:32.928880 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-89c85546f-nz7r8_4603e9ff-6845-42b9-9f15-d3c794ffea74/console/0.log" Jul 28 18:56:32.965813 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:32.965777 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-chwbs_8495b74a-e4fa-4aee-a599-a46e993f7455/download-server/0.log" Jul 28 18:56:33.583247 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:33.583214 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-2mpqv_96e9ed9f-ec57-4493-9bdf-5f86ee6d54b6/volume-data-source-validator/0.log" Jul 28 18:56:34.677786 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:34.677738 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ddcz_8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0/dns/0.log" Jul 28 18:56:34.701558 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:34.701529 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6ddcz_8f0cb6d3-97f2-44b3-aaef-c5b1aa3ff1f0/kube-rbac-proxy/0.log" Jul 28 18:56:34.918935 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:34.918864 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-mghpz_a3ba8fda-d419-4c09-880f-8a768f7cd1a0/dns-node-resolver/0.log" Jul 28 18:56:35.559851 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:35.559819 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rgqsk_98740ddb-0e74-4f8a-9e02-615ea70bc398/node-ca/0.log" Jul 28 18:56:36.784964 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:36.784935 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfd4f6bf9-cxdlb_d71a188f-e9d4-490c-902c-e25ef6dfb5b5/kube-auth-proxy/0.log" Jul 28 18:56:36.966440 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:36.966404 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tprs8_43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c/payload-processing/0.log" Jul 28 18:56:37.849558 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:37.849527 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kwltp_b5cfce10-c28f-463f-ab5e-0626d77afe0d/serve-healthcheck-canary/0.log" Jul 28 18:56:38.400310 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:38.400280 2575 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xtqp2/perf-node-gather-daemonset-jsdtv" Jul 28 18:56:38.571153 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:38.571125 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9pkg_51c0509b-69ea-4443-b397-6e7fb7313c19/kube-rbac-proxy/0.log" Jul 28 18:56:38.596624 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:38.596582 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9pkg_51c0509b-69ea-4443-b397-6e7fb7313c19/exporter/0.log" Jul 28 18:56:38.624311 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:38.624280 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l9pkg_51c0509b-69ea-4443-b397-6e7fb7313c19/extractor/0.log" Jul 28 18:56:41.879908 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:41.879881 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fb95c4cb9-kmp44_f65179d2-f023-42c2-8f23-94791e9afc77/manager/0.log" Jul 28 18:56:41.909091 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:41.909059 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5fb95c4cb9-kmp44_f65179d2-f023-42c2-8f23-94791e9afc77/copy-manifests/0.log" Jul 28 18:56:42.518245 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:42.518183 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfd4f6bf9-cxdlb_d71a188f-e9d4-490c-902c-e25ef6dfb5b5/kube-auth-proxy/0.log" Jul 28 18:56:42.707557 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:42.707522 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tprs8_43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c/payload-processing/0.log" Jul 28 18:56:43.320186 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:43.320079 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7cd6475d44-pf9b5_c51ec2af-2948-4ad8-ab2a-d5e1b6fc009b/manager/0.log" Jul 28 18:56:43.374767 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:43.374720 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-rnslk_641a8e7b-0e61-4f49-8ead-4eb6a9c143da/openshift-lws-operator/0.log" Jul 28 18:56:43.824410 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:43.824377 2575 scope.go:117] "RemoveContainer" containerID="2092e7a29e2f30068d2cdc5e4213b81ffd7a3ab751dcb093418af717d095bc19" Jul 28 18:56:43.837624 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:43.837602 2575 scope.go:117] "RemoveContainer" containerID="87db87915c843c42ce39dd4b69b652ce7bbc0575159dd2ba4e8e853b6f962edb" Jul 28 18:56:44.422361 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:44.422318 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7cfd4f6bf9-cxdlb_d71a188f-e9d4-490c-902c-e25ef6dfb5b5/kube-auth-proxy/0.log" Jul 28 18:56:44.562562 ip-10-0-135-147 kubenswrapper[2575]: I0728 18:56:44.562529 2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-tprs8_43bdcee5-23de-4dcf-899f-fa0ce2dc1d1c/payload-processing/0.log"