Jul 24 15:38:59.100728 ip-10-0-143-210 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 24 15:38:59.100742 ip-10-0-143-210 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 24 15:38:59.100752 ip-10-0-143-210 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 24 15:38:59.101109 ip-10-0-143-210 systemd[1]: Failed to start Kubernetes Kubelet. Jul 24 15:39:09.344251 ip-10-0-143-210 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 24 15:39:09.344270 ip-10-0-143-210 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 916481bc31cc40faaa42a28a22accf17 -- Jul 24 15:41:07.264317 ip-10-0-143-210 systemd[1]: Starting Kubernetes Kubelet... Jul 24 15:41:07.648977 ip-10-0-143-210 kubenswrapper[2543]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:41:07.648977 ip-10-0-143-210 kubenswrapper[2543]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 15:41:07.648977 ip-10-0-143-210 kubenswrapper[2543]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:41:07.648977 ip-10-0-143-210 kubenswrapper[2543]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 15:41:07.648977 ip-10-0-143-210 kubenswrapper[2543]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 15:41:07.651417 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.651232 2543 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 15:41:07.655408 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655393 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:41:07.655408 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655408 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655412 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655414 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655420 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655424 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655427 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655430 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655433 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655435 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655438 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655441 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655443 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655445 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655448 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655451 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655454 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655456 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655459 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655463 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:41:07.655472 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655468 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655476 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655480 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655483 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655485 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655488 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655491 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655493 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655496 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655524 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655527 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655530 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655533 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655535 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655538 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655540 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655543 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655545 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655548 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:41:07.655920 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655550 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655553 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655555 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655560 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655563 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655567 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655570 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655572 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655575 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655578 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655580 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655583 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655585 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655588 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655591 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655595 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655598 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655600 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655603 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:41:07.656393 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655605 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655608 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655611 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655614 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655616 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655618 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655621 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655623 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655625 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655628 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655631 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655633 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655636 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655639 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655641 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655643 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655646 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655648 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655651 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655653 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:41:07.656869 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655656 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655658 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655661 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655663 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655666 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655668 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655671 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655673 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655676 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655679 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655681 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655684 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655686 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655689 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655691 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655694 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655696 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655699 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655701 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655704 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:41:07.657380 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.655706 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656406 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656412 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656416 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656419 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656422 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656425 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656428 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656431 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656433 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656436 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656438 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656441 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656444 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656447 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656450 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656452 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656455 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656457 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:41:07.657853 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656460 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656463 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656465 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656468 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656470 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656473 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656475 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656478 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656480 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656483 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656485 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656488 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656490 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656492 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656495 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656497 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656500 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656503 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656506 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656508 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656511 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:41:07.658372 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656514 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656516 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656519 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656521 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656524 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656527 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656530 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656532 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656536 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656539 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656541 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656544 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656546 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656549 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656552 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656554 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656557 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656559 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656562 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656564 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:41:07.658880 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656567 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656569 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656572 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656576 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656580 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656583 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656586 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656590 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656594 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656596 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656599 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656601 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656604 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656607 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656609 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656612 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656615 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656618 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:41:07.659395 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656621 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656624 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656626 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656629 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656631 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656634 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656636 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656639 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656642 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656644 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656646 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656649 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656651 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656654 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656658 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656661 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656664 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656667 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656670 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656672 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:41:07.659848 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656675 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.656677 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657070 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657076 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657079 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657083 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657086 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657089 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657092 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657094 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657097 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657100 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657105 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657108 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657111 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657115 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657117 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657120 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657124 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657128 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:41:07.660342 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657131 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657134 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657137 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657140 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657142 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657145 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657147 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657150 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657152 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657155 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657157 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657161 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657163 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657166 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657168 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657171 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657173 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657176 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657179 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657182 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:41:07.660820 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657184 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657187 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657189 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657192 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657194 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657197 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657199 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657202 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657204 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657206 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657209 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657211 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657214 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657216 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657218 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657221 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657223 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657226 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657228 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657231 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:41:07.661337 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657233 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657236 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657238 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657241 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657244 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657246 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657252 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657256 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657259 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657262 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657265 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657268 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657271 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657274 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657277 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657279 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657282 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657285 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657287 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:41:07.661824 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657290 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657293 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657295 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657298 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657300 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657302 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657305 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657307 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657310 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657312 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657315 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657317 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657320 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657322 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657325 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657327 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657329 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657332 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657334 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657339 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:41:07.662288 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657341 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.657343 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657421 2543 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657432 2543 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657441 2543 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657447 2543 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657454 2543 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657457 2543 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657461 2543 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657465 2543 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657468 2543 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657471 2543 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657474 2543 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657477 2543 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657480 2543 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657483 2543 flags.go:64] FLAG: --cgroup-root="" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657486 2543 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657489 2543 flags.go:64] FLAG: --client-ca-file="" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657492 2543 flags.go:64] FLAG: --cloud-provider="external" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657495 2543 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657498 2543 flags.go:64] FLAG: --cluster-domain="" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657501 2543 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657504 2543 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657507 2543 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 15:41:07.662764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657510 2543 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657514 2543 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657516 2543 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657519 2543 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657522 2543 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657525 2543 flags.go:64] FLAG: --contention-profiling="false" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657528 2543 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657532 2543 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657535 2543 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657538 2543 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657543 2543 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657546 2543 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657548 2543 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657551 2543 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657554 2543 flags.go:64] FLAG: --enable-server="true" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657557 2543 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657565 2543 flags.go:64] FLAG: --event-burst="100" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657568 2543 flags.go:64] FLAG: --event-qps="50" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657571 2543 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657574 2543 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657577 2543 flags.go:64] FLAG: --eviction-hard="" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657581 2543 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657583 2543 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657587 2543 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657590 2543 flags.go:64] FLAG: --eviction-soft="" Jul 24 15:41:07.663398 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657592 2543 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657595 2543 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657598 2543 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657601 2543 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657604 2543 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657606 2543 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657609 2543 flags.go:64] FLAG: --feature-gates="" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657612 2543 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657618 2543 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657621 2543 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657624 2543 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657627 2543 flags.go:64] FLAG: --healthz-port="10248" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657629 2543 flags.go:64] FLAG: --help="false" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657632 2543 flags.go:64] FLAG: --hostname-override="ip-10-0-143-210.ec2.internal" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657637 2543 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657639 2543 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657642 2543 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657645 2543 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657649 2543 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657651 2543 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657654 2543 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657656 2543 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657659 2543 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 15:41:07.664002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657667 2543 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657670 2543 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657673 2543 flags.go:64] FLAG: --kube-reserved="" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657676 2543 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657679 2543 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657681 2543 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657684 2543 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657687 2543 flags.go:64] FLAG: --lock-file="" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657690 2543 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657693 2543 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657696 2543 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657702 2543 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657704 2543 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657707 2543 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657710 2543 flags.go:64] FLAG: --logging-format="text" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657713 2543 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657716 2543 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657719 2543 flags.go:64] FLAG: --manifest-url="" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657722 2543 flags.go:64] FLAG: --manifest-url-header="" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657730 2543 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657733 2543 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657737 2543 flags.go:64] FLAG: --max-pods="110" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657739 2543 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657744 2543 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657747 2543 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 15:41:07.665002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657750 2543 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657753 2543 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657756 2543 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657758 2543 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657765 2543 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657768 2543 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657771 2543 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657774 2543 flags.go:64] FLAG: --pod-cidr="" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657782 2543 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657787 2543 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657790 2543 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657793 2543 flags.go:64] FLAG: --pods-per-core="0" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657796 2543 flags.go:64] FLAG: --port="10250" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657798 2543 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657801 2543 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-087d7f217fc01f533" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657804 2543 flags.go:64] FLAG: --qos-reserved="" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657807 2543 flags.go:64] FLAG: --read-only-port="10255" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657810 2543 flags.go:64] FLAG: --register-node="true" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657813 2543 flags.go:64] FLAG: --register-with-taints="" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657816 2543 flags.go:64] FLAG: --registry-burst="10" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657819 2543 flags.go:64] FLAG: --registry-qps="5" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657822 2543 flags.go:64] FLAG: --reserved-cpus="" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657824 2543 flags.go:64] FLAG: --reserved-memory="" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657828 2543 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657831 2543 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 15:41:07.666098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657834 2543 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657837 2543 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657840 2543 flags.go:64] FLAG: --runonce="false" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657842 2543 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657845 2543 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657850 2543 flags.go:64] FLAG: --seccomp-default="false" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657853 2543 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657856 2543 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657859 2543 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657862 2543 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657865 2543 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657868 2543 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657870 2543 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657873 2543 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657876 2543 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657879 2543 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657887 2543 flags.go:64] FLAG: --system-cgroups="" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657890 2543 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657895 2543 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657897 2543 flags.go:64] FLAG: --tls-cert-file="" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657900 2543 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657908 2543 flags.go:64] FLAG: --tls-min-version="" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657911 2543 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657913 2543 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657916 2543 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 15:41:07.666793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657919 2543 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657922 2543 flags.go:64] FLAG: --v="2" Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657925 2543 flags.go:64] FLAG: --version="false" Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657929 2543 flags.go:64] FLAG: --vmodule="" Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657932 2543 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.657935 2543 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658080 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658087 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658090 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658092 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658095 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658097 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658101 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658105 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658108 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658111 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658115 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658119 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658122 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658125 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:41:07.667683 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658128 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658131 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658134 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658138 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658144 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658148 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658150 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658153 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658155 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658158 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658160 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658163 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658165 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658168 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658170 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658172 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658175 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658177 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658180 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658183 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:41:07.668621 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658186 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658189 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658191 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658193 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658198 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658200 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658203 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658205 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658207 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658210 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658212 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658215 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658217 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658220 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658222 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658224 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658227 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658230 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658232 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658235 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:41:07.669220 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658237 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658240 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658242 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658245 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658247 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658249 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658252 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658255 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658257 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658259 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658262 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658265 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658268 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658270 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658273 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658275 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658279 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658281 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658284 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658286 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:41:07.669762 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658289 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658291 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658294 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658296 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658299 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658301 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658303 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658306 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658308 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658310 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658314 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658316 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658318 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658321 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658323 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658326 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658329 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658331 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658334 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658336 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:41:07.670438 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658339 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658341 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658344 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658347 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.658350 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.659076 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.666248 2543 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.666266 2543 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.667092 2543 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.667106 2543 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667192 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667200 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667206 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667210 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667215 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:41:07.671284 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667220 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667224 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667228 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667232 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667237 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667241 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667246 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667251 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667255 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667259 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667263 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667275 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667280 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667284 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667288 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667292 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667296 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667300 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667304 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:41:07.671767 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667308 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667312 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667316 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667320 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667327 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667334 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667339 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667344 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667350 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667357 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667362 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667367 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667373 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667378 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667382 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667387 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667391 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667396 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667400 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:41:07.672386 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667406 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667410 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667415 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667420 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667424 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667428 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667432 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667436 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667440 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667444 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667448 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667453 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667457 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667460 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667465 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667469 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667473 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667478 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667482 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667486 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:41:07.672878 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667491 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667495 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667499 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667503 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667507 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667511 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667515 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667519 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667523 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667527 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667531 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667535 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667539 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667543 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667547 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667554 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667560 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667564 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667569 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667573 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:41:07.673397 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667578 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667582 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667586 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667590 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667594 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667598 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667602 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667606 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667610 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667615 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667619 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667623 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667627 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667631 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667636 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:41:07.673926 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667640 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.667647 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667856 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667865 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667870 2543 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667874 2543 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667879 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667884 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667889 2543 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667893 2543 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667897 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667901 2543 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667905 2543 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667910 2543 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 15:41:07.674376 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667915 2543 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667919 2543 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667922 2543 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667926 2543 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667931 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667935 2543 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667940 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667944 2543 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667948 2543 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667952 2543 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667956 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667960 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667965 2543 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667968 2543 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667972 2543 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667977 2543 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667981 2543 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667985 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667989 2543 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667993 2543 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 15:41:07.674721 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.667997 2543 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668001 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668005 2543 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668009 2543 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668036 2543 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668041 2543 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668047 2543 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668053 2543 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668057 2543 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668062 2543 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668066 2543 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668070 2543 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668074 2543 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668078 2543 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668083 2543 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668087 2543 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668091 2543 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668095 2543 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668099 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 15:41:07.675308 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668103 2543 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668107 2543 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668111 2543 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668115 2543 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668120 2543 feature_gate.go:328] unrecognized feature gate: Example Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668124 2543 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668129 2543 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668134 2543 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668139 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668143 2543 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668147 2543 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668151 2543 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668155 2543 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668159 2543 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668163 2543 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668167 2543 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668171 2543 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668175 2543 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668179 2543 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668185 2543 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 15:41:07.675914 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668190 2543 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668196 2543 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668200 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668204 2543 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668208 2543 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668212 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668217 2543 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668221 2543 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668225 2543 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668229 2543 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668236 2543 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668241 2543 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668245 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668251 2543 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668255 2543 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668259 2543 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668264 2543 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668267 2543 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668271 2543 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 15:41:07.676747 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668276 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668280 2543 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668284 2543 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668288 2543 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668292 2543 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668296 2543 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668300 2543 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668304 2543 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:07.668308 2543 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.668316 2543 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.668459 2543 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.672100 2543 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.673417 2543 server.go:1019] "Starting client certificate rotation" Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.673523 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:41:07.677564 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.674308 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 15:41:07.677919 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.676235 2543 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:41:07.683192 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.683170 2543 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 15:41:07.695837 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.695820 2543 log.go:25] "Validated CRI v1 runtime API" Jul 24 15:41:07.697847 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.697831 2543 log.go:25] "Validated CRI v1 image API" Jul 24 15:41:07.698781 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.698758 2543 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 15:41:07.703783 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.703766 2543 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 c56c6a1c-9071-4a3b-9ed5-becf7d16ea26:/dev/nvme0n1p4 d4f66671-b0f4-40ae-a739-8f490814fc77:/dev/nvme0n1p3] Jul 24 15:41:07.703838 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.703784 2543 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 15:41:07.708758 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.708645 2543 manager.go:216] Machine: {Timestamp:2026-07-24 15:41:07.707317779 +0000 UTC m=+0.327450669 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3117078 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22672f9c2151c8bcf8f26172fd6171 SystemUUID:ec22672f-9c21-51c8-bcf8-f26172fd6171 BootID:916481bc-31cc-40fa-aa42-a28a22accf17 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:79:f7:b8:43:51 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:79:f7:b8:43:51 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:16:d5:9b:7f:70 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 15:41:07.708758 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.708756 2543 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 15:41:07.708862 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.708829 2543 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 15:41:07.711848 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.711824 2543 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 15:41:07.711981 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.711850 2543 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 15:41:07.712578 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.712561 2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:41:07.712610 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.712585 2543 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 15:41:07.712610 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.712597 2543 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 15:41:07.712610 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.712606 2543 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:41:07.713279 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.713269 2543 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 15:41:07.714050 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.714041 2543 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 15:41:07.715995 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.715983 2543 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:41:07.716121 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716111 2543 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 15:41:07.716158 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716142 2543 kubelet.go:486] "Attempting to sync node with API server" Jul 24 15:41:07.716189 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716160 2543 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 15:41:07.716189 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716177 2543 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 15:41:07.716189 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716187 2543 kubelet.go:398] "Adding apiserver pod source" Jul 24 15:41:07.716269 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716194 2543 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 15:41:07.716586 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716575 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:41:07.716632 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.716624 2543 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 15:41:07.719230 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.719216 2543 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 15:41:07.720874 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.720861 2543 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 15:41:07.720917 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.720892 2543 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 15:41:07.722312 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722299 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722316 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722323 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722329 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722337 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722343 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722351 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722357 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722363 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722368 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722377 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 15:41:07.722392 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.722382 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 15:41:07.723201 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.723191 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 15:41:07.723201 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.723201 2543 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 15:41:07.726846 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.726802 2543 server.go:1268] "Started kubelet" Jul 24 15:41:07.726922 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.726834 2543 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 15:41:07.726958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.726886 2543 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 15:41:07.727051 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.727000 2543 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 15:41:07.727374 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.727338 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 15:41:07.727489 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.727426 2543 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 15:41:07.727634 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.727613 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 15:41:07.727892 ip-10-0-143-210 systemd[1]: Started Kubernetes Kubelet. Jul 24 15:41:07.728904 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.728882 2543 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 15:41:07.729357 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.729344 2543 server.go:310] "Adding debug handlers to kubelet server" Jul 24 15:41:07.733273 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.733256 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x4psj" Jul 24 15:41:07.735083 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.734191 2543 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-210.ec2.internal.18c5441123b8a2aa default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-210.ec2.internal,UID:ip-10-0-143-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-210.ec2.internal,},FirstTimestamp:2026-07-24 15:41:07.726705322 +0000 UTC m=+0.346838199,LastTimestamp:2026-07-24 15:41:07.726705322 +0000 UTC m=+0.346838199,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-210.ec2.internal,}" Jul 24 15:41:07.735172 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.735100 2543 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 15:41:07.735798 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.735785 2543 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 15:41:07.735841 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.735804 2543 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 15:41:07.736001 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.735984 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:07.736452 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736439 2543 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 15:41:07.736545 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736478 2543 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 15:41:07.736627 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736612 2543 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 15:41:07.736704 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736611 2543 factory.go:55] Registering systemd factory Jul 24 15:41:07.736704 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736646 2543 factory.go:223] Registration of the systemd container factory successfully Jul 24 15:41:07.736771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736737 2543 reconstruct.go:97] "Volume reconstruction finished" Jul 24 15:41:07.736771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736745 2543 reconciler.go:29] "Reconciler: start to sync state" Jul 24 15:41:07.736967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736954 2543 factory.go:153] Registering CRI-O factory Jul 24 15:41:07.736967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.736968 2543 factory.go:223] Registration of the crio container factory successfully Jul 24 15:41:07.737132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.737033 2543 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 15:41:07.737132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.737057 2543 factory.go:103] Registering Raw factory Jul 24 15:41:07.737132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.737071 2543 manager.go:1196] Started watching for new ooms in manager Jul 24 15:41:07.737451 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.737436 2543 manager.go:349] Starting recovery of all containers Jul 24 15:41:07.742174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.742148 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-x4psj" Jul 24 15:41:07.745068 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.745044 2543 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 15:41:07.745163 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.745140 2543 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 15:41:07.746038 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.745981 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 15:41:07.746817 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.746801 2543 manager.go:354] Recovery completed Jul 24 15:41:07.752869 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.752856 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:07.757965 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.757949 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:07.758075 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.757976 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:07.758075 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.757986 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:07.758552 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.758537 2543 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 15:41:07.758552 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.758553 2543 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 15:41:07.758687 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.758557 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:07.758687 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.758570 2543 state_mem.go:36] "Initialized new in-memory state store" Jul 24 15:41:07.760113 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.760097 2543 policy_none.go:49] "None policy: Start" Jul 24 15:41:07.760227 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.760217 2543 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 15:41:07.760316 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.760306 2543 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 15:41:07.761135 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.761116 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:07.761202 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.761153 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:07.761202 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.761167 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:07.761406 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.761387 2543 policy_none.go:47] "Start" Jul 24 15:41:07.801720 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.801551 2543 manager.go:337] "Starting Device Plugin manager" Jul 24 15:41:07.801814 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.801744 2543 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 15:41:07.801814 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.801757 2543 server.go:87] "Starting device plugin registration server" Jul 24 15:41:07.801969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.801959 2543 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 15:41:07.802009 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.801970 2543 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 15:41:07.802103 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.802088 2543 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 15:41:07.802182 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.802166 2543 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 15:41:07.802182 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.802179 2543 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 15:41:07.802731 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.802710 2543 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 15:41:07.802787 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.802759 2543 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:07.902251 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.902200 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:07.903287 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.903160 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:07.903287 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.903186 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:07.903287 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.903195 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:07.903287 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.903221 2543 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-143-210.ec2.internal" Jul 24 15:41:07.910985 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.910971 2543 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-143-210.ec2.internal" Jul 24 15:41:07.911067 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.910992 2543 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-210.ec2.internal\": node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:07.920319 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.920304 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:07.948256 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.948237 2543 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 15:41:07.948318 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.948275 2543 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 15:41:07.948318 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.948294 2543 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 15:41:07.948401 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:07.948322 2543 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 15:41:07.950329 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:07.950313 2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:41:08.020531 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.020498 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:08.048952 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.048932 2543 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal"] Jul 24 15:41:08.049006 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.048988 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:08.049863 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.049848 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:08.049929 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.049876 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:08.049929 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.049890 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:08.051139 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051127 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:08.051261 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051247 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.051305 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051292 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:08.051892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051878 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:08.051962 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051900 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:08.051962 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051915 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:08.051962 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051928 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:08.051962 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051946 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:08.051962 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.051955 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:08.052915 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.052903 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.052961 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.052927 2543 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 15:41:08.053608 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.053594 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientMemory" Jul 24 15:41:08.053666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.053613 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 15:41:08.053666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.053627 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeHasSufficientPID" Jul 24 15:41:08.077725 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.077704 2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.081953 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.081936 2543 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-210.ec2.internal\" not found" node="ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.120898 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.120875 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:08.139090 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.139068 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c3aeac3e09da0231a6a8f7c66c51509-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"8c3aeac3e09da0231a6a8f7c66c51509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.139166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.139094 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c3aeac3e09da0231a6a8f7c66c51509-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"8c3aeac3e09da0231a6a8f7c66c51509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.139166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.139111 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43749135b7c22609bc973c10a709b9e9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"43749135b7c22609bc973c10a709b9e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.221247 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.221174 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:08.239612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.239588 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c3aeac3e09da0231a6a8f7c66c51509-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"8c3aeac3e09da0231a6a8f7c66c51509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.239699 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.239621 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c3aeac3e09da0231a6a8f7c66c51509-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"8c3aeac3e09da0231a6a8f7c66c51509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.239699 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.239645 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43749135b7c22609bc973c10a709b9e9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"43749135b7c22609bc973c10a709b9e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.239834 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.239696 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/43749135b7c22609bc973c10a709b9e9-config\") pod \"kube-apiserver-proxy-ip-10-0-143-210.ec2.internal\" (UID: \"43749135b7c22609bc973c10a709b9e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.239834 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.239701 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/8c3aeac3e09da0231a6a8f7c66c51509-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"8c3aeac3e09da0231a6a8f7c66c51509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.239834 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.239701 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8c3aeac3e09da0231a6a8f7c66c51509-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal\" (UID: \"8c3aeac3e09da0231a6a8f7c66c51509\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.321785 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.321745 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:08.381249 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.381223 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.385310 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.385167 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 24 15:41:08.422787 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.422761 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:08.523299 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:08.523233 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:08.552975 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:08.552950 2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:41:09.711878 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:09.711855 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:09.761377 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:09.761358 2543 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 15:41:09.761779 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:09.761472 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:41:09.761779 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:09.761520 2543 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 15:41:09.812875 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:09.812838 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:09.839937 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:09.839907 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 15:36:08 +0000 UTC" deadline="2028-04-04 00:18:23.247436376 +0000 UTC" Jul 24 15:41:09.839937 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:09.839934 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14864h37m13.407505029s" Jul 24 15:41:09.876825 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:09.876778 2543 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:41:09.913845 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:09.913822 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:10.014371 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.014344 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:10.022741 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:10.022697 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3aeac3e09da0231a6a8f7c66c51509.slice/crio-ebf754cbe32fd7d27cfb0f58f981700f793321dcbdb008e5722708dd8bfad8bb WatchSource:0}: Error finding container ebf754cbe32fd7d27cfb0f58f981700f793321dcbdb008e5722708dd8bfad8bb: Status 404 returned error can't find the container with id ebf754cbe32fd7d27cfb0f58f981700f793321dcbdb008e5722708dd8bfad8bb Jul 24 15:41:10.022970 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:10.022944 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43749135b7c22609bc973c10a709b9e9.slice/crio-ca5d95d484ee7affd681299000571dda92d1d6806c2886753433cf72ecc65abb WatchSource:0}: Error finding container ca5d95d484ee7affd681299000571dda92d1d6806c2886753433cf72ecc65abb: Status 404 returned error can't find the container with id ca5d95d484ee7affd681299000571dda92d1d6806c2886753433cf72ecc65abb Jul 24 15:41:10.027686 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.027650 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:41:10.040336 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.040285 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" event={"ID":"43749135b7c22609bc973c10a709b9e9","Type":"ContainerStarted","Data":"ca5d95d484ee7affd681299000571dda92d1d6806c2886753433cf72ecc65abb"} Jul 24 15:41:10.041519 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.041500 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"8c3aeac3e09da0231a6a8f7c66c51509","Type":"ContainerStarted","Data":"ebf754cbe32fd7d27cfb0f58f981700f793321dcbdb008e5722708dd8bfad8bb"} Jul 24 15:41:10.114692 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.114658 2543 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-143-210.ec2.internal\" not found" Jul 24 15:41:10.143273 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.143220 2543 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:41:10.224967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.224944 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" Jul 24 15:41:10.240431 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.240409 2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:41:10.241513 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.241501 2543 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" Jul 24 15:41:10.249768 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.249752 2543 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 15:41:10.645134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.645106 2543 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:41:10.805891 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.805861 2543 apiserver.go:52] "Watching apiserver" Jul 24 15:41:10.811229 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.811197 2543 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 15:41:10.812367 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.812342 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-gcjzl","openshift-ovn-kubernetes/ovnkube-node-qtjgf","kube-system/global-pull-secret-syncer-4shzl","kube-system/konnectivity-agent-zs6lc","kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal","openshift-cluster-node-tuning-operator/tuned-xzn8p","openshift-image-registry/node-ca-knhkb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal","openshift-multus/network-metrics-daemon-kmrg6","openshift-network-diagnostics/network-check-target-lkzq7","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n","openshift-dns/node-resolver-7clmb","openshift-multus/multus-additional-cni-plugins-lfd8c","openshift-multus/multus-mcz5j"] Jul 24 15:41:10.814797 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.814777 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.816918 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.816897 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.817048 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.817016 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-576wp\"" Jul 24 15:41:10.817407 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.817386 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.817610 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.817596 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 15:41:10.817796 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.817781 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.818886 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.818870 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.818968 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.818893 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 15:41:10.819016 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.818869 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-47phx\"" Jul 24 15:41:10.819437 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.819186 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.819437 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.819249 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:10.819437 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.819263 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.821293 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.821272 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.823722 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.823698 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.824698 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.824325 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 15:41:10.824698 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.824456 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 15:41:10.824698 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.824488 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-59nmh\"" Jul 24 15:41:10.825025 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.825014 2543 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 15:41:10.825945 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.825926 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.826046 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.826033 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.826296 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.826276 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.826529 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.826509 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xxnrf\"" Jul 24 15:41:10.826682 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.826646 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 15:41:10.826767 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.826751 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 15:41:10.830480 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.830458 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:10.830813 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.830583 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:10.832295 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.832276 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.832389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.832375 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.832711 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.832693 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7mk27\"" Jul 24 15:41:10.834508 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.834489 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.835900 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.835882 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-sys\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.835987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.835913 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-lib-modules\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.835987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.835938 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-tuned\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.835987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.835961 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4nb\" (UniqueName: \"kubernetes.io/projected/026ec0c8-5f84-448d-91f8-3176793d2c28-kube-api-access-6w4nb\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.836112 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.835994 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhx9c\" (UniqueName: \"kubernetes.io/projected/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-kube-api-access-rhx9c\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.836112 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836044 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-cnibin\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836112 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836066 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-os-release\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836112 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836090 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-cni-multus\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836290 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836114 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-daemon-config\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836290 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836137 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-modprobe-d\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.836290 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836158 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-serviceca\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.836290 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836192 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-socket-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.836290 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836225 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-k8s-cni-cncf-io\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836290 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836254 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-etc-kubernetes\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836318 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysctl-conf\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836337 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-run\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836356 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-var-lib-kubelet\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836369 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-host\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836381 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-host\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836394 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-cni-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836423 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836456 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-device-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836480 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836494 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-sys-fs\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836513 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjtwj\" (UniqueName: \"kubernetes.io/projected/b9f87229-5da1-4bc4-86da-10afe507b953-kube-api-access-tjtwj\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836529 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b79ecb11-0e25-4a2a-8292-215db1faa83d-kubelet-config\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836542 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/777aa0dd-5fc1-4460-b509-b5090e3b07ff-cni-binary-copy\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.836553 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836573 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-socket-dir-parent\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836607 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-multus-certs\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836645 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-systemd\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836666 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026ec0c8-5f84-448d-91f8-3176793d2c28-tmp\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836686 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-system-cni-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836727 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-registration-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836757 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b06a7c14-2007-4bf4-b601-d19eda6ac6f2-agent-certs\") pod \"konnectivity-agent-zs6lc\" (UID: \"b06a7c14-2007-4bf4-b601-d19eda6ac6f2\") " pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836785 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-hostroot\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836806 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-conf-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836835 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysconfig\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836826 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836859 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysctl-d\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836885 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b79ecb11-0e25-4a2a-8292-215db1faa83d-dbus\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836908 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836911 2543 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836929 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b06a7c14-2007-4bf4-b601-d19eda6ac6f2-konnectivity-ca\") pod \"konnectivity-agent-zs6lc\" (UID: \"b06a7c14-2007-4bf4-b601-d19eda6ac6f2\") " pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836952 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-netns\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.836982 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-kubelet\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.837005 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-kubernetes\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.837656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.837027 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 15:41:10.837656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.837030 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-cni-bin\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.837062 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6qpq\" (UniqueName: \"kubernetes.io/projected/777aa0dd-5fc1-4460-b509-b5090e3b07ff-kube-api-access-j6qpq\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.837656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.837093 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-knbwm\"" Jul 24 15:41:10.837656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.837296 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.839283 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.839264 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:10.839377 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.839342 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:10.839792 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.839775 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 15:41:10.839880 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.839783 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 15:41:10.840038 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.840019 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 15:41:10.840105 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.840067 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9qw9q\"" Jul 24 15:41:10.840334 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.840314 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 15:41:10.840542 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.840524 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.840657 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.840585 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.841662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.841641 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:10.843962 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.843939 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-hj2m4\"" Jul 24 15:41:10.844059 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.843971 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 15:41:10.844122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.844060 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:10.844659 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.844193 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.844659 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.844200 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.846137 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.846117 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 15:41:10.846842 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.846813 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 15:41:10.847037 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.847018 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5qw2l\"" Jul 24 15:41:10.867840 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.867822 2543 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-n76dg" Jul 24 15:41:10.876771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.876749 2543 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-n76dg" Jul 24 15:41:10.926341 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.926287 2543 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 15:41:10.937233 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937200 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.937359 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937243 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-conf-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.937359 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937268 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-systemd-units\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.937359 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937290 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-node-log\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.937359 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937298 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-conf-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.937359 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937313 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:10.937359 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937351 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-netns\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937385 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-kubernetes\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937416 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-netns\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937445 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-log-socket\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937460 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-systemd\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937483 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6qpq\" (UniqueName: \"kubernetes.io/projected/777aa0dd-5fc1-4460-b509-b5090e3b07ff-kube-api-access-j6qpq\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937507 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4nb\" (UniqueName: \"kubernetes.io/projected/026ec0c8-5f84-448d-91f8-3176793d2c28-kube-api-access-6w4nb\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937524 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-kubernetes\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937540 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhx9c\" (UniqueName: \"kubernetes.io/projected/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-kube-api-access-rhx9c\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937604 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-kubelet\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.937666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937632 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-os-release\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937730 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-cni-multus\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937767 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-socket-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937794 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-env-overrides\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937798 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-cni-multus\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937820 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-etc-kubernetes\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937844 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-run\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937866 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-host\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937888 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-host\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937888 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-etc-kubernetes\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937924 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937943 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-socket-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937950 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937953 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-host\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937978 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-slash\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937994 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-host\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938003 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-ovn\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938008 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.938022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.937996 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-run\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938028 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/777aa0dd-5fc1-4460-b509-b5090e3b07ff-cni-binary-copy\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938048 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-socket-dir-parent\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938050 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-etc-selinux\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938078 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-socket-dir-parent\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938101 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6bhz\" (UniqueName: \"kubernetes.io/projected/78dc9795-927b-461e-821f-a2d682083d9f-kube-api-access-r6bhz\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938128 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f0e84-4122-475c-bc3d-a2094890f618-tmp-dir\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938144 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2xz7\" (UniqueName: \"kubernetes.io/projected/8f7f0e84-4122-475c-bc3d-a2094890f618-kube-api-access-k2xz7\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938160 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938176 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78dc9795-927b-461e-821f-a2d682083d9f-ovn-node-metrics-cert\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938190 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s67rw\" (UniqueName: \"kubernetes.io/projected/b137edfb-74d6-490f-a7f7-77eb793f6411-kube-api-access-s67rw\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938211 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cnibin\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938226 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938274 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jhm9\" (UniqueName: \"kubernetes.io/projected/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-kube-api-access-4jhm9\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938319 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b06a7c14-2007-4bf4-b601-d19eda6ac6f2-agent-certs\") pod \"konnectivity-agent-zs6lc\" (UID: \"b06a7c14-2007-4bf4-b601-d19eda6ac6f2\") " pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938362 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-hostroot\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938458 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysconfig\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.938867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938475 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-hostroot\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938512 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysctl-d\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938540 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938576 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysconfig\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938584 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-ovnkube-script-lib\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938611 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/777aa0dd-5fc1-4460-b509-b5090e3b07ff-cni-binary-copy\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938611 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-system-cni-dir\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938679 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b79ecb11-0e25-4a2a-8292-215db1faa83d-dbus\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938663 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysctl-d\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938732 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938784 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b06a7c14-2007-4bf4-b601-d19eda6ac6f2-konnectivity-ca\") pod \"konnectivity-agent-zs6lc\" (UID: \"b06a7c14-2007-4bf4-b601-d19eda6ac6f2\") " pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938785 2543 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938845 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b79ecb11-0e25-4a2a-8292-215db1faa83d-dbus\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.938878 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938893 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-kubelet\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938956 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-kubelet\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:10.938954 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:11.438921302 +0000 UTC m=+2.970508908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:10.939691 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.938984 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-sys\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939112 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-lib-modules\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939131 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-var-lib-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939255 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-lib-modules\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939285 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-cni-bin\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939309 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-tuned\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939336 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-cni-bin\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939430 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b06a7c14-2007-4bf4-b601-d19eda6ac6f2-konnectivity-ca\") pod \"konnectivity-agent-zs6lc\" (UID: \"b06a7c14-2007-4bf4-b601-d19eda6ac6f2\") " pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939512 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-cni-netd\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939536 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-ovnkube-config\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939594 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939646 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f7f0e84-4122-475c-bc3d-a2094890f618-hosts-file\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939744 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-sys\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939735 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-cnibin\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939763 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-var-lib-cni-bin\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939810 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-os-release\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939877 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-daemon-config\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939883 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-cnibin\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.940680 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.939907 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-modprobe-d\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940027 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-modprobe-d\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940039 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-os-release\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940078 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-serviceca\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940122 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940161 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-k8s-cni-cncf-io\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940186 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysctl-conf\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940231 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-var-lib-kubelet\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940259 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b137edfb-74d6-490f-a7f7-77eb793f6411-iptables-alerter-script\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940284 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-cni-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940308 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-device-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940363 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-sys-fs\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940381 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjtwj\" (UniqueName: \"kubernetes.io/projected/b9f87229-5da1-4bc4-86da-10afe507b953-kube-api-access-tjtwj\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940397 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940413 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b79ecb11-0e25-4a2a-8292-215db1faa83d-kubelet-config\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940429 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-multus-certs\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940443 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-systemd\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.941499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940459 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026ec0c8-5f84-448d-91f8-3176793d2c28-tmp\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940474 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-run-netns\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940474 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-daemon-config\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940479 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-serviceca\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940551 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b79ecb11-0e25-4a2a-8292-215db1faa83d-kubelet-config\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940574 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-etc-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940601 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-sys-fs\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940610 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-var-lib-kubelet\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940621 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-device-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940621 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-k8s-cni-cncf-io\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940654 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6njdk\" (UniqueName: \"kubernetes.io/projected/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-kube-api-access-6njdk\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940662 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-multus-cni-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940673 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-systemd\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940679 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-system-cni-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940703 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-sysctl-conf\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940716 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-registration-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940725 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-system-cni-dir\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.942370 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940751 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/777aa0dd-5fc1-4460-b509-b5090e3b07ff-host-run-multus-certs\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.943166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940754 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:10.943166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940776 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/b9f87229-5da1-4bc4-86da-10afe507b953-registration-dir\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.943166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.940790 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b137edfb-74d6-490f-a7f7-77eb793f6411-host-slash\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:10.943166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.942546 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/026ec0c8-5f84-448d-91f8-3176793d2c28-etc-tuned\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.943166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.942744 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b06a7c14-2007-4bf4-b601-d19eda6ac6f2-agent-certs\") pod \"konnectivity-agent-zs6lc\" (UID: \"b06a7c14-2007-4bf4-b601-d19eda6ac6f2\") " pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:10.943929 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.943905 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/026ec0c8-5f84-448d-91f8-3176793d2c28-tmp\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.946308 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.946282 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6qpq\" (UniqueName: \"kubernetes.io/projected/777aa0dd-5fc1-4460-b509-b5090e3b07ff-kube-api-access-j6qpq\") pod \"multus-mcz5j\" (UID: \"777aa0dd-5fc1-4460-b509-b5090e3b07ff\") " pod="openshift-multus/multus-mcz5j" Jul 24 15:41:10.946890 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.946797 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4nb\" (UniqueName: \"kubernetes.io/projected/026ec0c8-5f84-448d-91f8-3176793d2c28-kube-api-access-6w4nb\") pod \"tuned-xzn8p\" (UID: \"026ec0c8-5f84-448d-91f8-3176793d2c28\") " pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:10.948151 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.947983 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjtwj\" (UniqueName: \"kubernetes.io/projected/b9f87229-5da1-4bc4-86da-10afe507b953-kube-api-access-tjtwj\") pod \"aws-ebs-csi-driver-node-l8f8n\" (UID: \"b9f87229-5da1-4bc4-86da-10afe507b953\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:10.949079 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:10.949062 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhx9c\" (UniqueName: \"kubernetes.io/projected/a7066765-5a23-4d0f-ac2d-ef3d0173cd65-kube-api-access-rhx9c\") pod \"node-ca-knhkb\" (UID: \"a7066765-5a23-4d0f-ac2d-ef3d0173cd65\") " pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:11.041961 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.041930 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-run-netns\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.041970 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-etc-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.041998 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-run-netns\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042005 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6njdk\" (UniqueName: \"kubernetes.io/projected/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-kube-api-access-6njdk\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:11.042134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042032 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042083 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-run-ovn-kubernetes\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042247 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-etc-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042282 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b137edfb-74d6-490f-a7f7-77eb793f6411-host-slash\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.042389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042323 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.042389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042350 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-systemd-units\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042373 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-node-log\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042400 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042423 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-log-socket\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042420 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b137edfb-74d6-490f-a7f7-77eb793f6411-host-slash\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042432 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-systemd-units\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042439 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-systemd\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042473 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-systemd\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042480 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-kubelet\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042496 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-log-socket\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042509 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-os-release\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042509 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-kubelet\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042537 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-env-overrides\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042547 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-node-log\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042575 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-slash\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042585 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-os-release\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042607 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-ovn\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042625 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6bhz\" (UniqueName: \"kubernetes.io/projected/78dc9795-927b-461e-821f-a2d682083d9f-kube-api-access-r6bhz\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042634 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-slash\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.042662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042640 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f0e84-4122-475c-bc3d-a2094890f618-tmp-dir\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042669 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2xz7\" (UniqueName: \"kubernetes.io/projected/8f7f0e84-4122-475c-bc3d-a2094890f618-kube-api-access-k2xz7\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042696 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042699 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-ovn\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042725 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78dc9795-927b-461e-821f-a2d682083d9f-ovn-node-metrics-cert\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042768 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s67rw\" (UniqueName: \"kubernetes.io/projected/b137edfb-74d6-490f-a7f7-77eb793f6411-kube-api-access-s67rw\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042794 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cnibin\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042818 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042879 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f7f0e84-4122-475c-bc3d-a2094890f618-tmp-dir\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042879 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jhm9\" (UniqueName: \"kubernetes.io/projected/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-kube-api-access-4jhm9\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042887 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cnibin\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042934 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042967 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-ovnkube-script-lib\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.042976 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043016 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-system-cni-dir\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043053 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-run-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043064 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-var-lib-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.043481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043083 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-env-overrides\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043096 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-cni-bin\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043126 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-cni-netd\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043160 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-ovnkube-config\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043171 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-var-lib-openvswitch\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043194 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043223 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-system-cni-dir\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043224 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f7f0e84-4122-475c-bc3d-a2094890f618-hosts-file\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043277 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8f7f0e84-4122-475c-bc3d-a2094890f618-hosts-file\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043324 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-cni-bin\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043343 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.043354 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043277 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.043407 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:41:11.543390147 +0000 UTC m=+3.074977752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043440 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b137edfb-74d6-490f-a7f7-77eb793f6411-iptables-alerter-script\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043546 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-cni-netd\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043540 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.044415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043153 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/78dc9795-927b-461e-821f-a2d682083d9f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.045072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043947 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.045072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.043981 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-ovnkube-config\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.045072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.044192 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/78dc9795-927b-461e-821f-a2d682083d9f-ovnkube-script-lib\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.045072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.044200 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b137edfb-74d6-490f-a7f7-77eb793f6411-iptables-alerter-script\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.045072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.044432 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-cni-binary-copy\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.045940 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.045912 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/78dc9795-927b-461e-821f-a2d682083d9f-ovn-node-metrics-cert\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.048409 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.048375 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:11.048528 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.048410 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:11.048528 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.048433 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:11.048528 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.048518 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:11.548497583 +0000 UTC m=+3.080085196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:11.051124 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.051097 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6njdk\" (UniqueName: \"kubernetes.io/projected/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-kube-api-access-6njdk\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:11.051521 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.051495 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s67rw\" (UniqueName: \"kubernetes.io/projected/b137edfb-74d6-490f-a7f7-77eb793f6411-kube-api-access-s67rw\") pod \"iptables-alerter-gcjzl\" (UID: \"b137edfb-74d6-490f-a7f7-77eb793f6411\") " pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.051881 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.051855 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2xz7\" (UniqueName: \"kubernetes.io/projected/8f7f0e84-4122-475c-bc3d-a2094890f618-kube-api-access-k2xz7\") pod \"node-resolver-7clmb\" (UID: \"8f7f0e84-4122-475c-bc3d-a2094890f618\") " pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.052433 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.052410 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jhm9\" (UniqueName: \"kubernetes.io/projected/f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd-kube-api-access-4jhm9\") pod \"multus-additional-cni-plugins-lfd8c\" (UID: \"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd\") " pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.052682 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.052654 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6bhz\" (UniqueName: \"kubernetes.io/projected/78dc9795-927b-461e-821f-a2d682083d9f-kube-api-access-r6bhz\") pod \"ovnkube-node-qtjgf\" (UID: \"78dc9795-927b-461e-821f-a2d682083d9f\") " pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.126922 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.126880 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-knhkb" Jul 24 15:41:11.135263 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.135238 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" Jul 24 15:41:11.146013 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.145994 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:11.154671 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.154643 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:11.164473 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.164454 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mcz5j" Jul 24 15:41:11.170115 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.170095 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" Jul 24 15:41:11.177762 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.177713 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" Jul 24 15:41:11.190354 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.190335 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gcjzl" Jul 24 15:41:11.195011 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.194991 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7clmb" Jul 24 15:41:11.447155 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.447075 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:11.447301 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.447232 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:11.447301 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.447292 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:12.447276368 +0000 UTC m=+3.978863958 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:11.548145 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.548114 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:11.548269 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.548215 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:11.548269 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.548267 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:41:12.548252213 +0000 UTC m=+4.079839805 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:11.648948 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.648911 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:11.649122 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.649094 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:11.649122 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.649114 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:11.649240 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.649129 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:11.649240 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:11.649204 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:12.64918722 +0000 UTC m=+4.180774814 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:11.880510 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.880458 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:36:10 +0000 UTC" deadline="2028-03-21 14:07:23.039305797 +0000 UTC" Jul 24 15:41:11.880510 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:11.880506 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14542h26m11.158803352s" Jul 24 15:41:11.917818 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.917786 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7066765_5a23_4d0f_ac2d_ef3d0173cd65.slice/crio-70abafa0194f4b9819661709520cc1b9846c7ea4e28c2eb6fda9bf1837238807 WatchSource:0}: Error finding container 70abafa0194f4b9819661709520cc1b9846c7ea4e28c2eb6fda9bf1837238807: Status 404 returned error can't find the container with id 70abafa0194f4b9819661709520cc1b9846c7ea4e28c2eb6fda9bf1837238807 Jul 24 15:41:11.919899 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.919877 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod026ec0c8_5f84_448d_91f8_3176793d2c28.slice/crio-77838b9b17790d3c5423033333a9b09b8f3b8567cf67ecb07c26c842d948f0f9 WatchSource:0}: Error finding container 77838b9b17790d3c5423033333a9b09b8f3b8567cf67ecb07c26c842d948f0f9: Status 404 returned error can't find the container with id 77838b9b17790d3c5423033333a9b09b8f3b8567cf67ecb07c26c842d948f0f9 Jul 24 15:41:11.920370 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.920352 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9f87229_5da1_4bc4_86da_10afe507b953.slice/crio-5d8b5f70e1e15249094bbe243359b509b8ee910db082b8f55fd435628eb94c61 WatchSource:0}: Error finding container 5d8b5f70e1e15249094bbe243359b509b8ee910db082b8f55fd435628eb94c61: Status 404 returned error can't find the container with id 5d8b5f70e1e15249094bbe243359b509b8ee910db082b8f55fd435628eb94c61 Jul 24 15:41:11.924962 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.924758 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb06a7c14_2007_4bf4_b601_d19eda6ac6f2.slice/crio-21ce30da27b764a15e934cca6d5a1d609bb8a42839e37f750f3db53f4b9693fb WatchSource:0}: Error finding container 21ce30da27b764a15e934cca6d5a1d609bb8a42839e37f750f3db53f4b9693fb: Status 404 returned error can't find the container with id 21ce30da27b764a15e934cca6d5a1d609bb8a42839e37f750f3db53f4b9693fb Jul 24 15:41:11.926581 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.926542 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7f0e84_4122_475c_bc3d_a2094890f618.slice/crio-69a3bae00a050c18a23a824eb977fe44ceb531daca419290f25ad3bfe386497d WatchSource:0}: Error finding container 69a3bae00a050c18a23a824eb977fe44ceb531daca419290f25ad3bfe386497d: Status 404 returned error can't find the container with id 69a3bae00a050c18a23a824eb977fe44ceb531daca419290f25ad3bfe386497d Jul 24 15:41:11.927810 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.927729 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb137edfb_74d6_490f_a7f7_77eb793f6411.slice/crio-5ce550631bf142bcb2958519524739862f0dc20af8b905123dc2fcd92c4ee94c WatchSource:0}: Error finding container 5ce550631bf142bcb2958519524739862f0dc20af8b905123dc2fcd92c4ee94c: Status 404 returned error can't find the container with id 5ce550631bf142bcb2958519524739862f0dc20af8b905123dc2fcd92c4ee94c Jul 24 15:41:11.929144 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.928804 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf819a2b1_b2c5_4b6a_b823_af2ef15ba1bd.slice/crio-1c8196dc0acdf1c80ca04f3668a3943076fefc7bbdc3868fb1f3fdd863c7228a WatchSource:0}: Error finding container 1c8196dc0acdf1c80ca04f3668a3943076fefc7bbdc3868fb1f3fdd863c7228a: Status 404 returned error can't find the container with id 1c8196dc0acdf1c80ca04f3668a3943076fefc7bbdc3868fb1f3fdd863c7228a Jul 24 15:41:11.930256 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:11.930217 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78dc9795_927b_461e_821f_a2d682083d9f.slice/crio-d67cb403e4f47fece97fb4e513bcc18c32d67b5cef8123e8ede4412f5c8d0f45 WatchSource:0}: Error finding container d67cb403e4f47fece97fb4e513bcc18c32d67b5cef8123e8ede4412f5c8d0f45: Status 404 returned error can't find the container with id d67cb403e4f47fece97fb4e513bcc18c32d67b5cef8123e8ede4412f5c8d0f45 Jul 24 15:41:12.037327 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.037175 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:12.037470 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.037228 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:12.037523 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.037476 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:12.037523 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.037504 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:12.045478 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.045452 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"d67cb403e4f47fece97fb4e513bcc18c32d67b5cef8123e8ede4412f5c8d0f45"} Jul 24 15:41:12.046785 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.046757 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerStarted","Data":"1c8196dc0acdf1c80ca04f3668a3943076fefc7bbdc3868fb1f3fdd863c7228a"} Jul 24 15:41:12.047935 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.047917 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gcjzl" event={"ID":"b137edfb-74d6-490f-a7f7-77eb793f6411","Type":"ContainerStarted","Data":"5ce550631bf142bcb2958519524739862f0dc20af8b905123dc2fcd92c4ee94c"} Jul 24 15:41:12.049160 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.049137 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7clmb" event={"ID":"8f7f0e84-4122-475c-bc3d-a2094890f618","Type":"ContainerStarted","Data":"69a3bae00a050c18a23a824eb977fe44ceb531daca419290f25ad3bfe386497d"} Jul 24 15:41:12.050258 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.050222 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zs6lc" event={"ID":"b06a7c14-2007-4bf4-b601-d19eda6ac6f2","Type":"ContainerStarted","Data":"21ce30da27b764a15e934cca6d5a1d609bb8a42839e37f750f3db53f4b9693fb"} Jul 24 15:41:12.051281 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.051256 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mcz5j" event={"ID":"777aa0dd-5fc1-4460-b509-b5090e3b07ff","Type":"ContainerStarted","Data":"c0736ca0f3281cf0e433398bd900173c798ba4f013af0fc53d10d7a092c7f2cd"} Jul 24 15:41:12.053192 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.053169 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" event={"ID":"43749135b7c22609bc973c10a709b9e9","Type":"ContainerStarted","Data":"9b09d969e5b4266482a8be06c625d9df05df1e094d34e609833d22711ff94861"} Jul 24 15:41:12.054226 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.054208 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" event={"ID":"b9f87229-5da1-4bc4-86da-10afe507b953","Type":"ContainerStarted","Data":"5d8b5f70e1e15249094bbe243359b509b8ee910db082b8f55fd435628eb94c61"} Jul 24 15:41:12.055325 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.055306 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" event={"ID":"026ec0c8-5f84-448d-91f8-3176793d2c28","Type":"ContainerStarted","Data":"77838b9b17790d3c5423033333a9b09b8f3b8567cf67ecb07c26c842d948f0f9"} Jul 24 15:41:12.056438 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.056421 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knhkb" event={"ID":"a7066765-5a23-4d0f-ac2d-ef3d0173cd65","Type":"ContainerStarted","Data":"70abafa0194f4b9819661709520cc1b9846c7ea4e28c2eb6fda9bf1837238807"} Jul 24 15:41:12.067742 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.067691 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-210.ec2.internal" podStartSLOduration=2.067679674 podStartE2EDuration="2.067679674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:12.067114143 +0000 UTC m=+3.598701772" watchObservedRunningTime="2026-07-24 15:41:12.067679674 +0000 UTC m=+3.599267286" Jul 24 15:41:12.454948 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.454872 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:12.455098 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.454999 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:12.455098 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.455056 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:14.455037238 +0000 UTC m=+5.986624852 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:12.550495 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.550453 2543 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 15:41:12.555773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.555744 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:12.555920 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.555903 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:12.555987 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.555967 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:41:14.555948959 +0000 UTC m=+6.087536558 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:12.656706 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.656670 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:12.656887 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.656829 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:12.656887 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.656846 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:12.656887 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.656859 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:12.657048 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:12.656924 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:14.656905873 +0000 UTC m=+6.188493471 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:12.881503 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.881453 2543 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 15:36:10 +0000 UTC" deadline="2028-01-12 19:55:34.830818894 +0000 UTC" Jul 24 15:41:12.881503 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:12.881496 2543 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12892h14m21.949327378s" Jul 24 15:41:13.040549 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:13.040518 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:13.040736 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:13.040685 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:13.069294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:13.068616 2543 generic.go:356] "Generic (PLEG): container finished" podID="8c3aeac3e09da0231a6a8f7c66c51509" containerID="40fcacf03b87c844d7dc4a52e1fb60a960354e04b21a29f0b9fda78858d28fc3" exitCode=0 Jul 24 15:41:13.077964 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:13.077613 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"8c3aeac3e09da0231a6a8f7c66c51509","Type":"ContainerDied","Data":"40fcacf03b87c844d7dc4a52e1fb60a960354e04b21a29f0b9fda78858d28fc3"} Jul 24 15:41:14.037534 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:14.037478 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:14.038126 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.037623 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:14.038126 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:14.037697 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:14.038126 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.037769 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:14.083518 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:14.083450 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" event={"ID":"8c3aeac3e09da0231a6a8f7c66c51509","Type":"ContainerStarted","Data":"066c81d26663d1f8f859e5d1a62ee0e25c19dd6c408d25220284f428aca0bcf2"} Jul 24 15:41:14.472726 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:14.472071 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:14.472726 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.472262 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:14.472726 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.472323 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:18.472305749 +0000 UTC m=+10.003893346 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:14.573212 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:14.572595 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:14.573212 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.572798 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:14.573212 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.572863 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:41:18.572843449 +0000 UTC m=+10.104431046 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:14.674673 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:14.673996 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:14.674673 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.674213 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:14.674673 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.674235 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:14.674673 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.674248 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:14.674673 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:14.674305 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:18.674287497 +0000 UTC m=+10.205875094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:15.037705 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:15.037677 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:15.038165 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:15.037820 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:16.037987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:16.037954 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:16.038448 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:16.038083 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:16.038448 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:16.038141 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:16.038448 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:16.038206 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:17.037698 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:17.037619 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:17.037864 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:17.037774 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:18.037108 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:18.037064 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:18.037585 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.037195 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:18.037678 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:18.037653 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:18.037814 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.037751 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:18.508130 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:18.508090 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:18.508335 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.508250 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:18.508335 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.508306 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:26.508291189 +0000 UTC m=+18.039878779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:18.608937 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:18.608902 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:18.609099 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.609053 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:18.609185 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.609114 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:41:26.609096704 +0000 UTC m=+18.140684310 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:18.709958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:18.709920 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:18.710135 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.710110 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:18.710135 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.710132 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:18.710283 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.710143 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:18.710283 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:18.710188 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:26.710176371 +0000 UTC m=+18.241763976 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:19.039254 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:19.038782 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:19.039254 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:19.038906 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:20.037421 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:20.037392 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:20.037617 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:20.037503 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:20.037901 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:20.037880 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:20.037998 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:20.037977 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:21.037516 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:21.037480 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:21.037979 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:21.037632 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:22.037033 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:22.037003 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:22.037224 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:22.037000 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:22.037224 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:22.037131 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:22.037224 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:22.037190 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:23.037166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:23.037139 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:23.037582 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:23.037292 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:24.037408 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:24.037375 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:24.037807 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:24.037375 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:24.037807 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:24.037491 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:24.037807 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:24.037575 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:25.037409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:25.037374 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:25.037896 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:25.037518 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:26.038108 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:26.038073 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:26.038108 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:26.038114 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:26.038731 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.038223 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:26.038731 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.038358 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:26.571876 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:26.571823 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:26.572107 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.571978 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:26.572107 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.572049 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:41:42.572029221 +0000 UTC m=+34.103616811 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:26.672898 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:26.672860 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:26.673053 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.673019 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:26.673110 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.673090 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:41:42.67307134 +0000 UTC m=+34.204658941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:26.773356 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:26.773321 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:26.773538 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.773494 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:26.773538 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.773515 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:26.773538 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.773526 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:26.773721 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:26.773609 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:41:42.7735929 +0000 UTC m=+34.305180510 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:27.037762 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:27.037735 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:27.037931 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:27.037857 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:28.037362 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:28.037333 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:28.037869 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:28.037370 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:28.037869 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:28.037456 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:28.037869 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:28.037589 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:29.038812 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:29.038775 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:29.039314 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:29.038901 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:30.038015 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:30.037940 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:30.038176 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:30.037953 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:30.038176 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:30.038088 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:30.038176 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:30.038163 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:31.038113 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.037708 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:31.038827 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:31.038285 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:31.119353 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.119330 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"8db8c663f2134ab940b362485716a8808c20e233c468bd86cfaca18660cd7452"} Jul 24 15:41:31.119462 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.119446 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"1f4431a05b63e4c4bed374aa6037bdfd395259e47a90520c3a7755f53cab2be4"} Jul 24 15:41:31.119517 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.119505 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"22be5480f2cd2e82d40a55319a471ad8cef57baf6afab298078970fd9687e754"} Jul 24 15:41:31.119606 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.119552 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"c521c4798fbeeee761495d5ab5cc8ecd5fa9bb1905d4979e2a7f4a4fb17502ba"} Jul 24 15:41:31.120828 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.120805 2543 generic.go:356] "Generic (PLEG): container finished" podID="f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd" containerID="f08d50a2a8671e6eb25048c7c2578b98b0990249690dbef8a4f252b903362be9" exitCode=0 Jul 24 15:41:31.120909 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.120883 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerDied","Data":"f08d50a2a8671e6eb25048c7c2578b98b0990249690dbef8a4f252b903362be9"} Jul 24 15:41:31.122199 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.122179 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7clmb" event={"ID":"8f7f0e84-4122-475c-bc3d-a2094890f618","Type":"ContainerStarted","Data":"80062bd02a736130cb797fca95e7944f13c2a4935206dfb37051ab0771474139"} Jul 24 15:41:31.123472 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.123450 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zs6lc" event={"ID":"b06a7c14-2007-4bf4-b601-d19eda6ac6f2","Type":"ContainerStarted","Data":"0a2a717f70e02ec9bf3ce971308c3788fa7c5cde159cd10a7af5eafbcb45812c"} Jul 24 15:41:31.124770 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.124753 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mcz5j" event={"ID":"777aa0dd-5fc1-4460-b509-b5090e3b07ff","Type":"ContainerStarted","Data":"9a078f7f42f855b41d04aecd2f1563c2672686ec0fe782f0ea57dfc7f739e258"} Jul 24 15:41:31.126176 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.126149 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" event={"ID":"b9f87229-5da1-4bc4-86da-10afe507b953","Type":"ContainerStarted","Data":"f846412020743726fe321ed31694941075dc8fa77c7320c0320bd9d490e05715"} Jul 24 15:41:31.127913 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.127883 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" event={"ID":"026ec0c8-5f84-448d-91f8-3176793d2c28","Type":"ContainerStarted","Data":"9741c1796c22cec7e432a14d984cbda8e10b0b65d49767760b40ede0790733ac"} Jul 24 15:41:31.129055 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.129038 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-knhkb" event={"ID":"a7066765-5a23-4d0f-ac2d-ef3d0173cd65","Type":"ContainerStarted","Data":"9eda1e7348f7ff2a979c609095e00ad67edee28dfa0f642e7b1063f6403c3ad5"} Jul 24 15:41:31.142253 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.142219 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-210.ec2.internal" podStartSLOduration=21.142206303 podStartE2EDuration="21.142206303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:14.098768539 +0000 UTC m=+5.630356152" watchObservedRunningTime="2026-07-24 15:41:31.142206303 +0000 UTC m=+22.673793915" Jul 24 15:41:31.156584 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.156531 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-knhkb" podStartSLOduration=3.39554238 podStartE2EDuration="22.156519329s" totalImagesPullingTime="18.760976949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:31.156037085 +0000 UTC m=+22.687624699" watchObservedRunningTime="2026-07-24 15:41:31.156519329 +0000 UTC m=+22.688106941" Jul 24 15:41:31.171045 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.171009 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zs6lc" podStartSLOduration=3.416880734 podStartE2EDuration="22.170993272s" totalImagesPullingTime="18.754112538s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:31.170950951 +0000 UTC m=+22.702538592" watchObservedRunningTime="2026-07-24 15:41:31.170993272 +0000 UTC m=+22.702580883" Jul 24 15:41:31.189250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.189184 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7clmb" podStartSLOduration=3.4237184210000002 podStartE2EDuration="22.189172004s" totalImagesPullingTime="18.765453583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:31.188740926 +0000 UTC m=+22.720328539" watchObservedRunningTime="2026-07-24 15:41:31.189172004 +0000 UTC m=+22.720759617" Jul 24 15:41:31.207623 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.207537 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mcz5j" podStartSLOduration=3.390947045 podStartE2EDuration="22.207522885s" totalImagesPullingTime="18.81657584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:31.206886035 +0000 UTC m=+22.738473646" watchObservedRunningTime="2026-07-24 15:41:31.207522885 +0000 UTC m=+22.739110499" Jul 24 15:41:31.223518 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:31.223489 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xzn8p" podStartSLOduration=3.466209188 podStartE2EDuration="22.223479686s" totalImagesPullingTime="18.757270498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:31.223310185 +0000 UTC m=+22.754897797" watchObservedRunningTime="2026-07-24 15:41:31.223479686 +0000 UTC m=+22.755067299" Jul 24 15:41:32.037465 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.037445 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:32.037630 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:32.037537 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:32.037630 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.037592 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:32.037739 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:32.037669 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:32.138502 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.138216 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"7e299a9add0c9e3feb2e462bdf13bda40ef4d458a622557d87bb4b1b3b159b09"} Jul 24 15:41:32.138502 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.138310 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"19d0c78a6f764ed6aac18cbc96936909393b3a7579c3f4d0a82671c2296eeee0"} Jul 24 15:41:32.139948 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.139870 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gcjzl" event={"ID":"b137edfb-74d6-490f-a7f7-77eb793f6411","Type":"ContainerStarted","Data":"a1194272b9a2f6c462c07cd975557e21eec9f5fd754b6d2a85c9f39548cce3ed"} Jul 24 15:41:32.146891 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.146867 2543 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 15:41:32.157705 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.156684 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gcjzl" podStartSLOduration=4.40215884 podStartE2EDuration="23.156665893s" totalImagesPullingTime="18.754507053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:32.155655984 +0000 UTC m=+23.687243624" watchObservedRunningTime="2026-07-24 15:41:32.156665893 +0000 UTC m=+23.688253507" Jul 24 15:41:32.906927 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.906845 2543 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T15:41:32.146889008Z","UUID":"dcc01fed-04a4-4e0d-8a44-97cb60744b0f","Handler":null,"Name":"","Endpoint":""} Jul 24 15:41:32.908606 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.908582 2543 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 15:41:32.908743 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:32.908619 2543 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 15:41:33.039699 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:33.039669 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:33.039888 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:33.039806 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:33.144315 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:33.144273 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" event={"ID":"b9f87229-5da1-4bc4-86da-10afe507b953","Type":"ContainerStarted","Data":"ce4bbdd98035fcd7e476858b5dcc4d68527a4778c017485a2a9abe809930e575"} Jul 24 15:41:33.970541 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:33.970509 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:33.971225 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:33.971201 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:34.037898 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.037868 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:34.038103 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.037868 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:34.038103 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:34.037972 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:34.038103 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:34.038037 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:34.149464 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.149424 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" event={"ID":"b9f87229-5da1-4bc4-86da-10afe507b953","Type":"ContainerStarted","Data":"b2fcef34a9ce4d783b73803cf9e6e9619ed3bb6f2643b2ab9a3020b175175925"} Jul 24 15:41:34.153433 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.153411 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"a4b668d2953b3ad3b46af6323de4fa8e56b3b6175ea0af1a93228d1e2588db18"} Jul 24 15:41:34.153728 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.153707 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:34.154095 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.154079 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zs6lc" Jul 24 15:41:34.166994 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:34.166936 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l8f8n" podStartSLOduration=4.129293905 podStartE2EDuration="25.166921566s" totalImagesPullingTime="21.037627661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:34.166294701 +0000 UTC m=+25.697882315" watchObservedRunningTime="2026-07-24 15:41:34.166921566 +0000 UTC m=+25.698509179" Jul 24 15:41:35.037636 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:35.037376 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:35.037829 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:35.037750 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:36.037892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.037859 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:36.037892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.037884 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:36.038451 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:36.037965 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:36.038451 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:36.038075 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:36.160664 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.160586 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" event={"ID":"78dc9795-927b-461e-821f-a2d682083d9f","Type":"ContainerStarted","Data":"cca8bc76e466aa1ed55aa56aa7ccc693be0eddca9b651ba103d7d264c75b8d5d"} Jul 24 15:41:36.160880 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.160861 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:36.160978 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.160937 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:36.162803 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.162773 2543 generic.go:356] "Generic (PLEG): container finished" podID="f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd" containerID="271a866311210fd1162acaebe93fd47136f0c9a537227ec52d582a665d6d7bd3" exitCode=0 Jul 24 15:41:36.162925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.162838 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerDied","Data":"271a866311210fd1162acaebe93fd47136f0c9a537227ec52d582a665d6d7bd3"} Jul 24 15:41:36.176550 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.176527 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:36.176826 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.176802 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:36.185744 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:36.185705 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" podStartSLOduration=8.367602306 podStartE2EDuration="27.185691911s" totalImagesPullingTime="18.818089605s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:36.184297189 +0000 UTC m=+27.715884802" watchObservedRunningTime="2026-07-24 15:41:36.185691911 +0000 UTC m=+27.717279559" Jul 24 15:41:37.040846 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.040671 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:37.041235 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:37.040925 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:37.165822 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.165785 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:41:37.886261 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.886225 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4shzl"] Jul 24 15:41:37.886501 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.886369 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:37.886501 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:37.886452 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:37.889601 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.889546 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kmrg6"] Jul 24 15:41:37.889757 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.889705 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:37.889833 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:37.889813 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:37.890097 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.890064 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lkzq7"] Jul 24 15:41:37.890179 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:37.890143 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:37.890237 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:37.890220 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:38.169712 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:38.169676 2543 generic.go:356] "Generic (PLEG): container finished" podID="f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd" containerID="e9df8030922533dfdbb241186181be4ab60bec88d80ca2515200c7eb4eef73da" exitCode=0 Jul 24 15:41:38.170173 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:38.169763 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerDied","Data":"e9df8030922533dfdbb241186181be4ab60bec88d80ca2515200c7eb4eef73da"} Jul 24 15:41:39.038702 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:39.038672 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:39.038858 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:39.038756 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:39.856197 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:39.855964 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7clmb_8f7f0e84-4122-475c-bc3d-a2094890f618/dns-node-resolver/0.log" Jul 24 15:41:40.037205 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:40.037182 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:40.037334 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:40.037221 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:40.037334 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:40.037286 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:40.037422 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:40.037392 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:40.175778 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:40.175744 2543 generic.go:356] "Generic (PLEG): container finished" podID="f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd" containerID="9ce6a0ce7a8a7327094349ea1eee4d5240599a5ad50c000e451e5c9cc8b4d793" exitCode=0 Jul 24 15:41:40.175976 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:40.175801 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerDied","Data":"9ce6a0ce7a8a7327094349ea1eee4d5240599a5ad50c000e451e5c9cc8b4d793"} Jul 24 15:41:40.833136 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:40.833108 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knhkb_a7066765-5a23-4d0f-ac2d-ef3d0173cd65/node-ca/0.log" Jul 24 15:41:41.040057 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:41.040031 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:41.040417 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:41.040122 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:42.037239 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:42.037207 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:42.037416 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:42.037257 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:42.037416 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.037357 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:42.037529 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.037485 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:42.590987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:42.590953 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:42.591421 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.591062 2543 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:42.591421 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.591116 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret podName:b79ecb11-0e25-4a2a-8292-215db1faa83d nodeName:}" failed. No retries permitted until 2026-07-24 15:42:14.591100009 +0000 UTC m=+66.122687601 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret") pod "global-pull-secret-syncer-4shzl" (UID: "b79ecb11-0e25-4a2a-8292-215db1faa83d") : object "kube-system"/"original-pull-secret" not registered Jul 24 15:41:42.691845 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:42.691808 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:42.692022 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.691956 2543 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:42.692081 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.692023 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs podName:7be2c164-3700-4df9-a9b6-2e9b0fc2db0e nodeName:}" failed. No retries permitted until 2026-07-24 15:42:14.692003396 +0000 UTC m=+66.223590989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs") pod "network-metrics-daemon-kmrg6" (UID: "7be2c164-3700-4df9-a9b6-2e9b0fc2db0e") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 15:41:42.792580 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:42.792529 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:42.792742 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.792703 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 15:41:42.792742 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.792733 2543 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 15:41:42.792824 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.792747 2543 projected.go:196] Error preparing data for projected volume kube-api-access-qzrnt for pod openshift-network-diagnostics/network-check-target-lkzq7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:42.792863 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:42.792827 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt podName:56de8441-6f73-4c87-8011-60959bd15945 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:14.792806931 +0000 UTC m=+66.324394542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-qzrnt" (UniqueName: "kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt") pod "network-check-target-lkzq7" (UID: "56de8441-6f73-4c87-8011-60959bd15945") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 15:41:43.037793 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:43.037761 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:43.037975 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:43.037877 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:44.037544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:44.037510 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:44.038133 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:44.037555 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:44.038133 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:44.037646 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:44.038133 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:44.037769 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:45.037687 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:45.037657 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:45.038211 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:45.037780 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:45.189391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:45.189357 2543 generic.go:356] "Generic (PLEG): container finished" podID="f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd" containerID="6be6a50ac52c2a3b09a11fd79a8510c66b274e10e11350a6ab53553837860ba8" exitCode=0 Jul 24 15:41:45.189550 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:45.189415 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerDied","Data":"6be6a50ac52c2a3b09a11fd79a8510c66b274e10e11350a6ab53553837860ba8"} Jul 24 15:41:46.037927 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:46.037897 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:46.038267 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:46.037897 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:46.038267 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:46.038002 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:46.038267 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:46.038075 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:46.194639 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:46.194612 2543 generic.go:356] "Generic (PLEG): container finished" podID="f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd" containerID="e62ef38e3feadbeb1056ccbc3b6ec3ebd00c2f68cdc6d4eb5a54b6a326f25ca1" exitCode=0 Jul 24 15:41:46.194795 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:46.194662 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerDied","Data":"e62ef38e3feadbeb1056ccbc3b6ec3ebd00c2f68cdc6d4eb5a54b6a326f25ca1"} Jul 24 15:41:47.040604 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:47.040554 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:47.041049 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:47.040663 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:47.199877 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:47.199843 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" event={"ID":"f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd","Type":"ContainerStarted","Data":"5de9cf502bc40b3d6001d67a50e4cef6726ad2dd46f333b0da816536a3109e3c"} Jul 24 15:41:47.222179 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:47.222128 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lfd8c" podStartSLOduration=8.362548043 podStartE2EDuration="38.222114186s" totalImagesPullingTime="29.859566143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:41:47.221755622 +0000 UTC m=+38.753343235" watchObservedRunningTime="2026-07-24 15:41:47.222114186 +0000 UTC m=+38.753701799" Jul 24 15:41:48.037301 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:48.037134 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:48.037445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:48.037201 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:48.037445 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:48.037370 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:48.037523 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:48.037451 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:49.040058 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:49.040030 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:49.040426 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:49.040112 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:50.037737 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:50.037706 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:50.037918 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:50.037716 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:50.037918 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:50.037799 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:50.038049 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:50.037908 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:51.038078 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:51.038043 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:51.038476 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:51.038158 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:52.037170 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:52.037141 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:52.037348 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:52.037249 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:52.037348 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:52.037295 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:52.037423 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:52.037393 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:53.038072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:53.038043 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:53.038451 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:53.038140 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:54.037976 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:54.037942 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:54.038202 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:54.037949 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:54.038202 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:54.038052 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:54.038202 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:54.038116 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:55.037671 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:55.037644 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:55.037833 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:55.037744 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:56.037791 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:56.037756 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:56.037791 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:56.037789 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:56.038241 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:56.037882 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:56.038241 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:56.037995 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:57.038021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:57.037978 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:57.038505 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:57.038117 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lkzq7" podUID="56de8441-6f73-4c87-8011-60959bd15945" Jul 24 15:41:58.037595 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.037542 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:41:58.037595 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.037577 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:41:58.037796 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:58.037682 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-kmrg6" podUID="7be2c164-3700-4df9-a9b6-2e9b0fc2db0e" Jul 24 15:41:58.037796 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:41:58.037767 2543 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-4shzl" podUID="b79ecb11-0e25-4a2a-8292-215db1faa83d" Jul 24 15:41:58.285542 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.285511 2543 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-143-210.ec2.internal" event="NodeReady" Jul 24 15:41:58.285914 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.285656 2543 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 15:41:58.337829 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.337757 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-7tr45"] Jul 24 15:41:58.348916 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.348898 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n9kqc"] Jul 24 15:41:58.349038 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.349023 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.351368 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.351348 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 15:41:58.351484 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.351434 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 15:41:58.351675 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.351658 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 15:41:58.351770 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.351722 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 15:41:58.351847 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.351796 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qtzxv\"" Jul 24 15:41:58.363032 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.363012 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n9kqc"] Jul 24 15:41:58.363126 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.363041 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7tr45"] Jul 24 15:41:58.363166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.363121 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.365608 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.365546 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 15:41:58.365608 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.365546 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-dz9d4\"" Jul 24 15:41:58.365608 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.365548 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 15:41:58.441626 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.441594 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-84hld"] Jul 24 15:41:58.462331 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.462308 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-84hld"] Jul 24 15:41:58.462467 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.462411 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.464686 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.464660 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 15:41:58.464686 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.464683 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 15:41:58.464855 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.464749 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-m8m7j\"" Jul 24 15:41:58.464855 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.464830 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 15:41:58.508439 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508414 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1e25853-89ba-4910-a9f7-d26e38ae3370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.508548 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508442 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1e25853-89ba-4910-a9f7-d26e38ae3370-data-volume\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.508548 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508462 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e87160a-8c3e-4880-acd9-409cf1515c2d-config-volume\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.508548 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508485 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e87160a-8c3e-4880-acd9-409cf1515c2d-tmp-dir\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.508548 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508504 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tv8\" (UniqueName: \"kubernetes.io/projected/b1e25853-89ba-4910-a9f7-d26e38ae3370-kube-api-access-w6tv8\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.508755 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508581 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e87160a-8c3e-4880-acd9-409cf1515c2d-metrics-tls\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.508755 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508631 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1e25853-89ba-4910-a9f7-d26e38ae3370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.508755 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508675 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x877t\" (UniqueName: \"kubernetes.io/projected/3e87160a-8c3e-4880-acd9-409cf1515c2d-kube-api-access-x877t\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.508869 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.508756 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1e25853-89ba-4910-a9f7-d26e38ae3370-crio-socket\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.609815 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.609759 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1e25853-89ba-4910-a9f7-d26e38ae3370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.609815 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.609787 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1e25853-89ba-4910-a9f7-d26e38ae3370-data-volume\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.609815 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.609812 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e87160a-8c3e-4880-acd9-409cf1515c2d-config-volume\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.610010 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.609830 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e87160a-8c3e-4880-acd9-409cf1515c2d-tmp-dir\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.610010 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.609851 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tv8\" (UniqueName: \"kubernetes.io/projected/b1e25853-89ba-4910-a9f7-d26e38ae3370-kube-api-access-w6tv8\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.610010 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.609975 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e87160a-8c3e-4880-acd9-409cf1515c2d-metrics-tls\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.610147 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610055 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1e25853-89ba-4910-a9f7-d26e38ae3370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.610147 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610086 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x877t\" (UniqueName: \"kubernetes.io/projected/3e87160a-8c3e-4880-acd9-409cf1515c2d-kube-api-access-x877t\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.610147 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610112 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc12ea4f-c891-41da-8174-b8f91d8b961e-cert\") pod \"ingress-canary-84hld\" (UID: \"bc12ea4f-c891-41da-8174-b8f91d8b961e\") " pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.610265 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610149 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjh8n\" (UniqueName: \"kubernetes.io/projected/bc12ea4f-c891-41da-8174-b8f91d8b961e-kube-api-access-tjh8n\") pod \"ingress-canary-84hld\" (UID: \"bc12ea4f-c891-41da-8174-b8f91d8b961e\") " pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.610265 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610173 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e87160a-8c3e-4880-acd9-409cf1515c2d-tmp-dir\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.610265 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610189 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1e25853-89ba-4910-a9f7-d26e38ae3370-crio-socket\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.610265 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610190 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b1e25853-89ba-4910-a9f7-d26e38ae3370-data-volume\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.610425 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610356 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b1e25853-89ba-4910-a9f7-d26e38ae3370-crio-socket\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.610509 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610486 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e87160a-8c3e-4880-acd9-409cf1515c2d-config-volume\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.610674 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.610658 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b1e25853-89ba-4910-a9f7-d26e38ae3370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.613786 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.613761 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b1e25853-89ba-4910-a9f7-d26e38ae3370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.613786 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.613777 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e87160a-8c3e-4880-acd9-409cf1515c2d-metrics-tls\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.617337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.617317 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x877t\" (UniqueName: \"kubernetes.io/projected/3e87160a-8c3e-4880-acd9-409cf1515c2d-kube-api-access-x877t\") pod \"dns-default-n9kqc\" (UID: \"3e87160a-8c3e-4880-acd9-409cf1515c2d\") " pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.617695 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.617675 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tv8\" (UniqueName: \"kubernetes.io/projected/b1e25853-89ba-4910-a9f7-d26e38ae3370-kube-api-access-w6tv8\") pod \"insights-runtime-extractor-7tr45\" (UID: \"b1e25853-89ba-4910-a9f7-d26e38ae3370\") " pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.659500 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.659481 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-7tr45" Jul 24 15:41:58.670653 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.670628 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n9kqc" Jul 24 15:41:58.710906 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.710846 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc12ea4f-c891-41da-8174-b8f91d8b961e-cert\") pod \"ingress-canary-84hld\" (UID: \"bc12ea4f-c891-41da-8174-b8f91d8b961e\") " pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.710906 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.710888 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjh8n\" (UniqueName: \"kubernetes.io/projected/bc12ea4f-c891-41da-8174-b8f91d8b961e-kube-api-access-tjh8n\") pod \"ingress-canary-84hld\" (UID: \"bc12ea4f-c891-41da-8174-b8f91d8b961e\") " pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.713312 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.713289 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc12ea4f-c891-41da-8174-b8f91d8b961e-cert\") pod \"ingress-canary-84hld\" (UID: \"bc12ea4f-c891-41da-8174-b8f91d8b961e\") " pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.719352 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.719324 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjh8n\" (UniqueName: \"kubernetes.io/projected/bc12ea4f-c891-41da-8174-b8f91d8b961e-kube-api-access-tjh8n\") pod \"ingress-canary-84hld\" (UID: \"bc12ea4f-c891-41da-8174-b8f91d8b961e\") " pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.771430 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.771397 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-84hld" Jul 24 15:41:58.868087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.868027 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-7tr45"] Jul 24 15:41:58.881063 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.881033 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n9kqc"] Jul 24 15:41:58.893840 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:58.893808 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e87160a_8c3e_4880_acd9_409cf1515c2d.slice/crio-bf53c02f859e8a828a3846dfd2151356b746c9b9a9c5455ae9aa5587ceecf1f0 WatchSource:0}: Error finding container bf53c02f859e8a828a3846dfd2151356b746c9b9a9c5455ae9aa5587ceecf1f0: Status 404 returned error can't find the container with id bf53c02f859e8a828a3846dfd2151356b746c9b9a9c5455ae9aa5587ceecf1f0 Jul 24 15:41:58.922677 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:58.922643 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-84hld"] Jul 24 15:41:58.925652 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:41:58.925631 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc12ea4f_c891_41da_8174_b8f91d8b961e.slice/crio-2fe9278f8080f6672dd54d30b672b33adb090a8eff2d43b9e102f00add8e0dd9 WatchSource:0}: Error finding container 2fe9278f8080f6672dd54d30b672b33adb090a8eff2d43b9e102f00add8e0dd9: Status 404 returned error can't find the container with id 2fe9278f8080f6672dd54d30b672b33adb090a8eff2d43b9e102f00add8e0dd9 Jul 24 15:41:59.038839 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.038811 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:41:59.041306 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.041288 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvmsc\"" Jul 24 15:41:59.041429 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.041329 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:41:59.041429 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.041292 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:41:59.226572 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.226524 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n9kqc" event={"ID":"3e87160a-8c3e-4880-acd9-409cf1515c2d","Type":"ContainerStarted","Data":"bf53c02f859e8a828a3846dfd2151356b746c9b9a9c5455ae9aa5587ceecf1f0"} Jul 24 15:41:59.227643 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.227616 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tr45" event={"ID":"b1e25853-89ba-4910-a9f7-d26e38ae3370","Type":"ContainerStarted","Data":"e021339b3bdbbc014d5544d6014baf7ac7aec7cc3e1c17a49039a0ed25d2d591"} Jul 24 15:41:59.228669 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:41:59.228647 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-84hld" event={"ID":"bc12ea4f-c891-41da-8174-b8f91d8b961e","Type":"ContainerStarted","Data":"2fe9278f8080f6672dd54d30b672b33adb090a8eff2d43b9e102f00add8e0dd9"} Jul 24 15:42:00.037237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:00.037202 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:42:00.037719 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:00.037659 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:42:00.041453 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:00.041427 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:42:00.041638 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:00.041618 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nh5k9\"" Jul 24 15:42:00.041710 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:00.041627 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:42:05.248850 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.248814 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n9kqc" event={"ID":"3e87160a-8c3e-4880-acd9-409cf1515c2d","Type":"ContainerStarted","Data":"0a3f1016f8523bb0548fd0e611a0d6281f397398ec706f6294ed2baf5693f2a3"} Jul 24 15:42:05.249330 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.248878 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n9kqc" event={"ID":"3e87160a-8c3e-4880-acd9-409cf1515c2d","Type":"ContainerStarted","Data":"384fd23bf9110d7bd322c7f2437cac4b883bd3a73013193448296437ef618914"} Jul 24 15:42:05.249330 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.248937 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n9kqc" Jul 24 15:42:05.250296 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.250275 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tr45" event={"ID":"b1e25853-89ba-4910-a9f7-d26e38ae3370","Type":"ContainerStarted","Data":"50c2f7f34561f21da60d481afd1ed5200b37e70da5bdf66c0dc8fa263ce360c1"} Jul 24 15:42:05.251422 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.251405 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-84hld" event={"ID":"bc12ea4f-c891-41da-8174-b8f91d8b961e","Type":"ContainerStarted","Data":"741e3b70e47812380db0512c35643020a5dc97ecadbc8857cb134e54c68acfad"} Jul 24 15:42:05.264201 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.264149 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-n9kqc" podStartSLOduration=1.426229706 podStartE2EDuration="7.264134334s" totalImagesPullingTime="5.837904628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:05.264035468 +0000 UTC m=+56.795623078" watchObservedRunningTime="2026-07-24 15:42:05.264134334 +0000 UTC m=+56.795721982" Jul 24 15:42:05.278018 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:05.277975 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-84hld" podStartSLOduration=1.471909255 podStartE2EDuration="7.27796091s" totalImagesPullingTime="5.806051655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:05.277811697 +0000 UTC m=+56.809399324" watchObservedRunningTime="2026-07-24 15:42:05.27796091 +0000 UTC m=+56.809548518" Jul 24 15:42:08.180777 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:08.180750 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-qtjgf" Jul 24 15:42:09.263602 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:09.263547 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tr45" event={"ID":"b1e25853-89ba-4910-a9f7-d26e38ae3370","Type":"ContainerStarted","Data":"afcf46599481af8341f3e367627804c48ba9179de289b470d29249957fe2378b"} Jul 24 15:42:11.141896 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.141861 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5"] Jul 24 15:42:11.155689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.155658 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5"] Jul 24 15:42:11.155846 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.155775 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:11.158847 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.158820 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 24 15:42:11.160161 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.160140 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-j7gr2\"" Jul 24 15:42:11.202377 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.202346 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fbfe7372-ca52-41b6-808c-40a9c0003f8e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-87vp5\" (UID: \"fbfe7372-ca52-41b6-808c-40a9c0003f8e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:11.303073 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.303037 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fbfe7372-ca52-41b6-808c-40a9c0003f8e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-87vp5\" (UID: \"fbfe7372-ca52-41b6-808c-40a9c0003f8e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:11.305835 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.305805 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/fbfe7372-ca52-41b6-808c-40a9c0003f8e-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-87vp5\" (UID: \"fbfe7372-ca52-41b6-808c-40a9c0003f8e\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:11.469480 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.469448 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:11.860997 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:11.860969 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5"] Jul 24 15:42:11.864207 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:11.864183 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfe7372_ca52_41b6_808c_40a9c0003f8e.slice/crio-b51a7c91a102700db58531acc049218bbd73dff0d540479fd59d511c8df295a9 WatchSource:0}: Error finding container b51a7c91a102700db58531acc049218bbd73dff0d540479fd59d511c8df295a9: Status 404 returned error can't find the container with id b51a7c91a102700db58531acc049218bbd73dff0d540479fd59d511c8df295a9 Jul 24 15:42:12.271776 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:12.271745 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" event={"ID":"fbfe7372-ca52-41b6-808c-40a9c0003f8e","Type":"ContainerStarted","Data":"b51a7c91a102700db58531acc049218bbd73dff0d540479fd59d511c8df295a9"} Jul 24 15:42:12.273547 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:12.273522 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-7tr45" event={"ID":"b1e25853-89ba-4910-a9f7-d26e38ae3370","Type":"ContainerStarted","Data":"51c43257d217fcf3d74b2a279b1067cb919a085302de79f28b57d61a9b4b7abe"} Jul 24 15:42:12.290291 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:12.290254 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-7tr45" podStartSLOduration=1.523962988 podStartE2EDuration="14.290241297s" totalImagesPullingTime="12.766278309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:12.28877297 +0000 UTC m=+63.820360598" watchObservedRunningTime="2026-07-24 15:42:12.290241297 +0000 UTC m=+63.821828910" Jul 24 15:42:13.280682 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:13.280636 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" event={"ID":"fbfe7372-ca52-41b6-808c-40a9c0003f8e","Type":"ContainerStarted","Data":"9498cd7fdaaf11fcf8392587b4bd2f16bf3d604e3944e34765b9bb2f39ce6f69"} Jul 24 15:42:13.295656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:13.295615 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" podStartSLOduration=1.208049983 podStartE2EDuration="2.29560011s" totalImagesPullingTime="1.087550127s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:13.29435577 +0000 UTC m=+64.825943390" watchObservedRunningTime="2026-07-24 15:42:13.29560011 +0000 UTC m=+64.827187723" Jul 24 15:42:14.161829 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.161796 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-dpctg"] Jul 24 15:42:14.163610 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.163596 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:14.165963 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.165933 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 24 15:42:14.166067 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.165996 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 24 15:42:14.167152 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.167132 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-7wtv8\"" Jul 24 15:42:14.174448 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.174426 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dpctg"] Jul 24 15:42:14.221418 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.221391 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmjz2\" (UniqueName: \"kubernetes.io/projected/3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7-kube-api-access-xmjz2\") pod \"downloads-7c97dc4555-dpctg\" (UID: \"3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7\") " pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:14.286844 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.286819 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:14.291857 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.291830 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-87vp5" Jul 24 15:42:14.322088 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.322056 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmjz2\" (UniqueName: \"kubernetes.io/projected/3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7-kube-api-access-xmjz2\") pod \"downloads-7c97dc4555-dpctg\" (UID: \"3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7\") " pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:14.330202 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.330174 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmjz2\" (UniqueName: \"kubernetes.io/projected/3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7-kube-api-access-xmjz2\") pod \"downloads-7c97dc4555-dpctg\" (UID: \"3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7\") " pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:14.472142 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.472114 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:14.593295 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.593265 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-dpctg"] Jul 24 15:42:14.597000 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:14.596974 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de9d359_9c0f_4c49_ba30_42b7bbd3fbb7.slice/crio-c86a8f6a081b1eb07cf1a5ed4abea6e20e1e22aabbce724c32169559e776e49d WatchSource:0}: Error finding container c86a8f6a081b1eb07cf1a5ed4abea6e20e1e22aabbce724c32169559e776e49d: Status 404 returned error can't find the container with id c86a8f6a081b1eb07cf1a5ed4abea6e20e1e22aabbce724c32169559e776e49d Jul 24 15:42:14.624813 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.624785 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:42:14.626966 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.626949 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 15:42:14.637855 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.637832 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b79ecb11-0e25-4a2a-8292-215db1faa83d-original-pull-secret\") pod \"global-pull-secret-syncer-4shzl\" (UID: \"b79ecb11-0e25-4a2a-8292-215db1faa83d\") " pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:42:14.725381 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.725290 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:42:14.727482 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.727466 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 15:42:14.738124 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.738101 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7be2c164-3700-4df9-a9b6-2e9b0fc2db0e-metrics-certs\") pod \"network-metrics-daemon-kmrg6\" (UID: \"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e\") " pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:42:14.754200 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.754180 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-4shzl" Jul 24 15:42:14.764502 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.764484 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nh5k9\"" Jul 24 15:42:14.773100 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.773081 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-kmrg6" Jul 24 15:42:14.826240 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.825969 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:42:14.828461 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.828442 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 15:42:14.838993 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.838972 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 15:42:14.850061 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.850040 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzrnt\" (UniqueName: \"kubernetes.io/projected/56de8441-6f73-4c87-8011-60959bd15945-kube-api-access-qzrnt\") pod \"network-check-target-lkzq7\" (UID: \"56de8441-6f73-4c87-8011-60959bd15945\") " pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:42:14.880717 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.880675 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-4shzl"] Jul 24 15:42:14.883392 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:14.883357 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79ecb11_0e25_4a2a_8292_215db1faa83d.slice/crio-2d12d13dc25f2ce0bf824af47217b216ffc97c9832f49ea0b35ecff6ec665c14 WatchSource:0}: Error finding container 2d12d13dc25f2ce0bf824af47217b216ffc97c9832f49ea0b35ecff6ec665c14: Status 404 returned error can't find the container with id 2d12d13dc25f2ce0bf824af47217b216ffc97c9832f49ea0b35ecff6ec665c14 Jul 24 15:42:14.899155 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.899133 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-kmrg6"] Jul 24 15:42:14.901790 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:14.901769 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be2c164_3700_4df9_a9b6_2e9b0fc2db0e.slice/crio-5aa35a6ca75cbc451bcaf38aa30434d5b1e65b0170c63616a3f9639b340526b1 WatchSource:0}: Error finding container 5aa35a6ca75cbc451bcaf38aa30434d5b1e65b0170c63616a3f9639b340526b1: Status 404 returned error can't find the container with id 5aa35a6ca75cbc451bcaf38aa30434d5b1e65b0170c63616a3f9639b340526b1 Jul 24 15:42:14.949753 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.949725 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hvmsc\"" Jul 24 15:42:14.957475 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:14.957457 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:42:15.079665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.079631 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lkzq7"] Jul 24 15:42:15.082278 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:15.082251 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56de8441_6f73_4c87_8011_60959bd15945.slice/crio-9eeff92e7bbb3fcaf58b903aa4e2383148258efad0e956f65e257ad7a6f90b6c WatchSource:0}: Error finding container 9eeff92e7bbb3fcaf58b903aa4e2383148258efad0e956f65e257ad7a6f90b6c: Status 404 returned error can't find the container with id 9eeff92e7bbb3fcaf58b903aa4e2383148258efad0e956f65e257ad7a6f90b6c Jul 24 15:42:15.239656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.239603 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-v5jqd"] Jul 24 15:42:15.242600 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.242553 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.245091 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.245069 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 24 15:42:15.245967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.245450 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rrvbn\"" Jul 24 15:42:15.245967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.245469 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 15:42:15.245967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.245528 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 15:42:15.245967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.245455 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 15:42:15.245967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.245782 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 24 15:42:15.256626 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.256571 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n9kqc" Jul 24 15:42:15.258538 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.258515 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-v5jqd"] Jul 24 15:42:15.291889 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.291860 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmrg6" event={"ID":"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e","Type":"ContainerStarted","Data":"5aa35a6ca75cbc451bcaf38aa30434d5b1e65b0170c63616a3f9639b340526b1"} Jul 24 15:42:15.293344 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.293318 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4shzl" event={"ID":"b79ecb11-0e25-4a2a-8292-215db1faa83d","Type":"ContainerStarted","Data":"2d12d13dc25f2ce0bf824af47217b216ffc97c9832f49ea0b35ecff6ec665c14"} Jul 24 15:42:15.294993 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.294963 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dpctg" event={"ID":"3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7","Type":"ContainerStarted","Data":"c86a8f6a081b1eb07cf1a5ed4abea6e20e1e22aabbce724c32169559e776e49d"} Jul 24 15:42:15.296297 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.296269 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lkzq7" event={"ID":"56de8441-6f73-4c87-8011-60959bd15945","Type":"ContainerStarted","Data":"9eeff92e7bbb3fcaf58b903aa4e2383148258efad0e956f65e257ad7a6f90b6c"} Jul 24 15:42:15.328980 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.328942 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.329134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.329101 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4zf7\" (UniqueName: \"kubernetes.io/projected/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-kube-api-access-s4zf7\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.329206 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.329155 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.329206 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.329188 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.429549 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.429513 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4zf7\" (UniqueName: \"kubernetes.io/projected/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-kube-api-access-s4zf7\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.429797 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.429587 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.429797 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.429621 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.429797 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.429670 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.430454 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.430430 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.431095 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:15.431071 2543 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 24 15:42:15.431209 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:15.431152 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-tls podName:02c7433e-a5ba-4ab1-aa74-ffd5435935ef nodeName:}" failed. No retries permitted until 2026-07-24 15:42:15.931133096 +0000 UTC m=+67.462720699 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-v5jqd" (UID: "02c7433e-a5ba-4ab1-aa74-ffd5435935ef") : secret "prometheus-operator-tls" not found Jul 24 15:42:15.433810 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.433784 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.441518 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.441493 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4zf7\" (UniqueName: \"kubernetes.io/projected/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-kube-api-access-s4zf7\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.932979 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.932941 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:15.946515 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:15.946451 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/02c7433e-a5ba-4ab1-aa74-ffd5435935ef-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-v5jqd\" (UID: \"02c7433e-a5ba-4ab1-aa74-ffd5435935ef\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:16.153907 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:16.153865 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" Jul 24 15:42:16.656365 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:16.656332 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-v5jqd"] Jul 24 15:42:16.838021 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:16.837938 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c7433e_a5ba_4ab1_aa74_ffd5435935ef.slice/crio-bfd624c647e96e10078efa2f4145b1e005865568174e6b9091c76a8a3783330d WatchSource:0}: Error finding container bfd624c647e96e10078efa2f4145b1e005865568174e6b9091c76a8a3783330d: Status 404 returned error can't find the container with id bfd624c647e96e10078efa2f4145b1e005865568174e6b9091c76a8a3783330d Jul 24 15:42:17.307947 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:17.307649 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmrg6" event={"ID":"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e","Type":"ContainerStarted","Data":"91ea998b1f7b7db163da1b15f8e22b8ae88f331563592c7ba11fc62e4bba9706"} Jul 24 15:42:17.307947 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:17.307705 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-kmrg6" event={"ID":"7be2c164-3700-4df9-a9b6-2e9b0fc2db0e","Type":"ContainerStarted","Data":"9c9acb15f1f635bcd00efde9883212d85eddb981d96fec698e625b5f6451ed7e"} Jul 24 15:42:17.309116 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:17.309049 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" event={"ID":"02c7433e-a5ba-4ab1-aa74-ffd5435935ef","Type":"ContainerStarted","Data":"bfd624c647e96e10078efa2f4145b1e005865568174e6b9091c76a8a3783330d"} Jul 24 15:42:21.332434 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.332347 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-4shzl" event={"ID":"b79ecb11-0e25-4a2a-8292-215db1faa83d","Type":"ContainerStarted","Data":"322131aea12aba1335f3361626128628fe8debae60529689834f355b36d91bf2"} Jul 24 15:42:21.334701 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.334668 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lkzq7" event={"ID":"56de8441-6f73-4c87-8011-60959bd15945","Type":"ContainerStarted","Data":"1f0fc9cacb944ebb92ed633b55608ab6977eaa785343b2ef2fd7673e84b13477"} Jul 24 15:42:21.334840 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.334739 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:42:21.337938 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.337912 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" event={"ID":"02c7433e-a5ba-4ab1-aa74-ffd5435935ef","Type":"ContainerStarted","Data":"9716b1e785fe259009a21881856373339102dff81d947cad87c8c6a8088c86a0"} Jul 24 15:42:21.338051 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.337957 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" event={"ID":"02c7433e-a5ba-4ab1-aa74-ffd5435935ef","Type":"ContainerStarted","Data":"070d2dc02ac8fd5f099c187797582bbf9283ee14a12f6418b535f891930ca3e5"} Jul 24 15:42:21.347245 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.347201 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-kmrg6" podStartSLOduration=70.381474413 podStartE2EDuration="1m12.347183198s" totalImagesPullingTime="1.965708785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:17.328692281 +0000 UTC m=+68.860279912" watchObservedRunningTime="2026-07-24 15:42:21.347183198 +0000 UTC m=+72.878770815" Jul 24 15:42:21.347654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.347590 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-4shzl" podStartSLOduration=66.128206529 podStartE2EDuration="1m12.34755198s" totalImagesPullingTime="6.219345451s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:21.346776439 +0000 UTC m=+72.878364054" watchObservedRunningTime="2026-07-24 15:42:21.34755198 +0000 UTC m=+72.879139593" Jul 24 15:42:21.363465 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.363427 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lkzq7" podStartSLOduration=66.36106618 podStartE2EDuration="1m12.363412786s" totalImagesPullingTime="6.002346606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:41:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:21.362602358 +0000 UTC m=+72.894189993" watchObservedRunningTime="2026-07-24 15:42:21.363412786 +0000 UTC m=+72.895000389" Jul 24 15:42:21.381056 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:21.381015 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-v5jqd" podStartSLOduration=2.156323039 podStartE2EDuration="6.38100236s" totalImagesPullingTime="4.224679321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:21.38076877 +0000 UTC m=+72.912356385" watchObservedRunningTime="2026-07-24 15:42:21.38100236 +0000 UTC m=+72.912589973" Jul 24 15:42:23.586353 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.586315 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8"] Jul 24 15:42:23.589049 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.589025 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.592075 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.592042 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 24 15:42:23.592384 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.592367 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-9gw2p\"" Jul 24 15:42:23.593316 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.593287 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 24 15:42:23.618535 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.618512 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8"] Jul 24 15:42:23.619742 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.619281 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v"] Jul 24 15:42:23.621622 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.621602 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.625985 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.625959 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 24 15:42:23.627258 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.627223 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 24 15:42:23.627494 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.627473 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pfq65\"" Jul 24 15:42:23.627643 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.627626 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 24 15:42:23.647925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.647893 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v"] Jul 24 15:42:23.660610 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.660590 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-sbs4d"] Jul 24 15:42:23.662621 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.662607 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.664914 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.664893 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 15:42:23.665030 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.664923 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-bk4xx\"" Jul 24 15:42:23.665030 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.664934 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 15:42:23.665030 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.664957 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 15:42:23.696901 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.696870 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.696993 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.696918 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2v9s\" (UniqueName: \"kubernetes.io/projected/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-api-access-t2v9s\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.696993 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.696948 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mbzr\" (UniqueName: \"kubernetes.io/projected/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-kube-api-access-8mbzr\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.696993 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.696980 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697008 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697044 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697066 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-wtmp\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697096 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-sys\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697124 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697149 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-textfile\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697172 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpgxx\" (UniqueName: \"kubernetes.io/projected/6c5cb812-a998-476f-9131-3d275a6981ef-kube-api-access-tpgxx\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.697207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697195 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-metrics-client-ca\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697233 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697272 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-root\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697317 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5cb812-a998-476f-9131-3d275a6981ef-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697340 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-tls\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697386 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697418 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.697660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.697455 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.797966 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.797929 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.797978 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.798148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798011 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2v9s\" (UniqueName: \"kubernetes.io/projected/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-api-access-t2v9s\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798039 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mbzr\" (UniqueName: \"kubernetes.io/projected/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-kube-api-access-8mbzr\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798071 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798101 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798135 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798157 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-wtmp\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:23.798161 2543 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798189 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-sys\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798219 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:23.798237 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-tls podName:6c5cb812-a998-476f-9131-3d275a6981ef nodeName:}" failed. No retries permitted until 2026-07-24 15:42:24.298213988 +0000 UTC m=+75.829801597 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-tls") pod "openshift-state-metrics-b8f49f5c6-8cpf8" (UID: "6c5cb812-a998-476f-9131-3d275a6981ef") : secret "openshift-state-metrics-tls" not found Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798263 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-textfile\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798291 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpgxx\" (UniqueName: \"kubernetes.io/projected/6c5cb812-a998-476f-9131-3d275a6981ef-kube-api-access-tpgxx\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798313 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-metrics-client-ca\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798337 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798336 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798356 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-root\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798391 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5cb812-a998-476f-9131-3d275a6981ef-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.798619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798421 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-tls\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.798619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798457 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.798619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798488 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798693 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.798852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.798743 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-wtmp\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799039 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799062 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-root\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799073 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-sys\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:23.799155 2543 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:23.799211 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-tls podName:9adb9c66-a5b2-44dc-aa3f-6efc22df5309 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:24.299193903 +0000 UTC m=+75.830781518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-tls") pod "node-exporter-sbs4d" (UID: "9adb9c66-a5b2-44dc-aa3f-6efc22df5309") : secret "node-exporter-tls" not found Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799391 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-accelerators-collector-config\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799480 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-textfile\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799597 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6c5cb812-a998-476f-9131-3d275a6981ef-metrics-client-ca\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.799771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799715 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.800282 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.799906 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-metrics-client-ca\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.801623 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.801578 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.801860 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.801831 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.801963 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.801922 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.802137 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.802114 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.814816 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.814790 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mbzr\" (UniqueName: \"kubernetes.io/projected/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-kube-api-access-8mbzr\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:23.815330 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.815310 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2v9s\" (UniqueName: \"kubernetes.io/projected/f264b9b3-6bc6-4dba-98d9-e29aca740f2d-kube-api-access-t2v9s\") pod \"kube-state-metrics-56d9f944b4-dg84v\" (UID: \"f264b9b3-6bc6-4dba-98d9-e29aca740f2d\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:23.815969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.815946 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpgxx\" (UniqueName: \"kubernetes.io/projected/6c5cb812-a998-476f-9131-3d275a6981ef-kube-api-access-tpgxx\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:23.932844 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:23.932778 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" Jul 24 15:42:24.076344 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.076321 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v"] Jul 24 15:42:24.078743 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:24.078708 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf264b9b3_6bc6_4dba_98d9_e29aca740f2d.slice/crio-44e40292f2273676a9de69d74b0a63cdcc5eb5c524ae65cd86b513d91364ad92 WatchSource:0}: Error finding container 44e40292f2273676a9de69d74b0a63cdcc5eb5c524ae65cd86b513d91364ad92: Status 404 returned error can't find the container with id 44e40292f2273676a9de69d74b0a63cdcc5eb5c524ae65cd86b513d91364ad92 Jul 24 15:42:24.285642 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.285610 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8474c7665c-5k796"] Jul 24 15:42:24.288075 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.288053 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.290584 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.290544 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:42:24.290696 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.290601 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:42:24.290696 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.290660 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vx7mq\"" Jul 24 15:42:24.290790 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.290746 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:42:24.290790 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.290771 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:42:24.290790 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.290787 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:42:24.298447 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.298430 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8474c7665c-5k796"] Jul 24 15:42:24.303047 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.303023 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-tls\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:24.303147 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.303090 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:24.305832 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.305807 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/9adb9c66-a5b2-44dc-aa3f-6efc22df5309-node-exporter-tls\") pod \"node-exporter-sbs4d\" (UID: \"9adb9c66-a5b2-44dc-aa3f-6efc22df5309\") " pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:24.305925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.305865 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/6c5cb812-a998-476f-9131-3d275a6981ef-openshift-state-metrics-tls\") pod \"openshift-state-metrics-b8f49f5c6-8cpf8\" (UID: \"6c5cb812-a998-476f-9131-3d275a6981ef\") " pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:24.351280 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.351249 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" event={"ID":"f264b9b3-6bc6-4dba-98d9-e29aca740f2d","Type":"ContainerStarted","Data":"44e40292f2273676a9de69d74b0a63cdcc5eb5c524ae65cd86b513d91364ad92"} Jul 24 15:42:24.404062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.403828 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfzk\" (UniqueName: \"kubernetes.io/projected/bdb263aa-d586-41c5-b00c-a459330e157d-kube-api-access-9rfzk\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.404062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.403885 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-oauth-config\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.404062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.403931 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-console-config\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.404062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.403984 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-service-ca\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.404062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.404025 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-serving-cert\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.404062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.404048 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-oauth-serving-cert\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.500179 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.500138 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" Jul 24 15:42:24.505450 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.505379 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-serving-cert\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.505450 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.505421 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-oauth-serving-cert\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.505659 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.505457 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfzk\" (UniqueName: \"kubernetes.io/projected/bdb263aa-d586-41c5-b00c-a459330e157d-kube-api-access-9rfzk\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.505659 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.505486 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-oauth-config\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.505659 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.505534 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-console-config\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.505659 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.505610 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-service-ca\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.506441 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.506406 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-service-ca\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.506802 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.506778 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-oauth-serving-cert\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.507269 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.507226 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-console-config\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.508693 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.508672 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-oauth-config\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.508798 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.508760 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-serving-cert\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.514952 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.514925 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfzk\" (UniqueName: \"kubernetes.io/projected/bdb263aa-d586-41c5-b00c-a459330e157d-kube-api-access-9rfzk\") pod \"console-8474c7665c-5k796\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.570894 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.570865 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-sbs4d" Jul 24 15:42:24.582470 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:24.582433 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9adb9c66_a5b2_44dc_aa3f_6efc22df5309.slice/crio-1e11e0331a566ea68379641ce4b1ab5d3996485404ac891608762c6b8979fd49 WatchSource:0}: Error finding container 1e11e0331a566ea68379641ce4b1ab5d3996485404ac891608762c6b8979fd49: Status 404 returned error can't find the container with id 1e11e0331a566ea68379641ce4b1ab5d3996485404ac891608762c6b8979fd49 Jul 24 15:42:24.598971 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.598944 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:24.652606 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.652543 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8"] Jul 24 15:42:24.655927 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:24.655894 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5cb812_a998_476f_9131_3d275a6981ef.slice/crio-fd22b719c4494d1fccc489958d8ab13cb3ed879e8b4c3d400cc6890237b5ffaf WatchSource:0}: Error finding container fd22b719c4494d1fccc489958d8ab13cb3ed879e8b4c3d400cc6890237b5ffaf: Status 404 returned error can't find the container with id fd22b719c4494d1fccc489958d8ab13cb3ed879e8b4c3d400cc6890237b5ffaf Jul 24 15:42:24.663501 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.663474 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:42:24.668243 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.667307 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.669978 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.669930 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 15:42:24.670079 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.669997 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zcdjr\"" Jul 24 15:42:24.671026 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.670242 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 15:42:24.671026 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.670406 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 15:42:24.671026 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.670454 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 15:42:24.671026 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.670714 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 15:42:24.672239 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.671607 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 15:42:24.672239 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.671632 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 15:42:24.672239 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.671837 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 15:42:24.672239 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.672081 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 15:42:24.685890 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.685801 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706643 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-out\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706686 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706723 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706756 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706777 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-web-config\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706805 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706836 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.707074 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.706888 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.708090 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.707776 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.708090 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.707824 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-volume\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.708090 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.707865 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.708090 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.707897 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx9cf\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-kube-api-access-qx9cf\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.708090 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.707922 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.761022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.760486 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8474c7665c-5k796"] Jul 24 15:42:24.767771 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:24.767738 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb263aa_d586_41c5_b00c_a459330e157d.slice/crio-b87c4047e58502bd5567030c6a4be0f1dac4615c8f41d069288dc94e0c587f26 WatchSource:0}: Error finding container b87c4047e58502bd5567030c6a4be0f1dac4615c8f41d069288dc94e0c587f26: Status 404 returned error can't find the container with id b87c4047e58502bd5567030c6a4be0f1dac4615c8f41d069288dc94e0c587f26 Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808619 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx9cf\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-kube-api-access-qx9cf\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808661 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808690 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-out\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808747 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808778 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808798 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-web-config\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808827 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808855 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808902 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808955 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.808987 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-volume\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.809773 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.809017 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.812354 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.811963 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.813897 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.813609 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.813897 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.813719 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-out\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.813897 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:24.813826 2543 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 24 15:42:24.813897 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:24.813847 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle podName:43f33209-9f25-4418-9c7b-e7eaf9adaa5c nodeName:}" failed. No retries permitted until 2026-07-24 15:42:25.313828172 +0000 UTC m=+76.845415780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c") : configmap references non-existent config key: ca-bundle.crt Jul 24 15:42:24.813897 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:24.813883 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls podName:43f33209-9f25-4418-9c7b-e7eaf9adaa5c nodeName:}" failed. No retries permitted until 2026-07-24 15:42:25.31386677 +0000 UTC m=+76.845454378 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c") : secret "alertmanager-main-tls" not found Jul 24 15:42:24.814193 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.813920 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.814193 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.814110 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.814764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.814680 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.815340 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.815271 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-web-config\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.816811 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.816742 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.817602 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.817520 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-volume\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.817875 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.817836 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:24.823230 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:24.823189 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx9cf\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-kube-api-access-qx9cf\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:25.314468 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.314429 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:25.314687 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.314520 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:25.315308 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.315282 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:25.317734 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.317687 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:25.358369 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.358325 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8474c7665c-5k796" event={"ID":"bdb263aa-d586-41c5-b00c-a459330e157d","Type":"ContainerStarted","Data":"b87c4047e58502bd5567030c6a4be0f1dac4615c8f41d069288dc94e0c587f26"} Jul 24 15:42:25.360725 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.360697 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" event={"ID":"6c5cb812-a998-476f-9131-3d275a6981ef","Type":"ContainerStarted","Data":"2708b838dd861b139da8d5888b7986bad1ee5c6c156c95402c060f11972df7eb"} Jul 24 15:42:25.360905 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.360886 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" event={"ID":"6c5cb812-a998-476f-9131-3d275a6981ef","Type":"ContainerStarted","Data":"3832c4d80a90a7269318281a4e86f900de138cb7e53de68fbb6e198f3a67d7c6"} Jul 24 15:42:25.361126 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.360924 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" event={"ID":"6c5cb812-a998-476f-9131-3d275a6981ef","Type":"ContainerStarted","Data":"fd22b719c4494d1fccc489958d8ab13cb3ed879e8b4c3d400cc6890237b5ffaf"} Jul 24 15:42:25.362537 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.362476 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbs4d" event={"ID":"9adb9c66-a5b2-44dc-aa3f-6efc22df5309","Type":"ContainerStarted","Data":"1e11e0331a566ea68379641ce4b1ab5d3996485404ac891608762c6b8979fd49"} Jul 24 15:42:25.582798 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:25.582711 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:42:26.580609 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.580439 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5c5d7468d-dg6nr"] Jul 24 15:42:26.586952 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.586030 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.589109 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.588758 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 24 15:42:26.589109 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.588843 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-27h2p9ac7riul\"" Jul 24 15:42:26.589852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.589492 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-hk9d9\"" Jul 24 15:42:26.589852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.589523 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 24 15:42:26.589852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.589717 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 24 15:42:26.589852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.589738 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 24 15:42:26.589852 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.589792 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 24 15:42:26.600461 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.600364 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:42:26.609172 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.609100 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c5d7468d-dg6nr"] Jul 24 15:42:26.611881 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:26.611846 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f33209_9f25_4418_9c7b_e7eaf9adaa5c.slice/crio-4953da384c4c547d6b8678dd3bc28f509ffc08cd9ddd6804de0f05f75482433f WatchSource:0}: Error finding container 4953da384c4c547d6b8678dd3bc28f509ffc08cd9ddd6804de0f05f75482433f: Status 404 returned error can't find the container with id 4953da384c4c547d6b8678dd3bc28f509ffc08cd9ddd6804de0f05f75482433f Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625205 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625268 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625303 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625367 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-grpc-tls\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625402 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625441 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnkk\" (UniqueName: \"kubernetes.io/projected/df152280-e06e-42df-b5ed-0ee10b3143ab-kube-api-access-6rnkk\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625478 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df152280-e06e-42df-b5ed-0ee10b3143ab-metrics-client-ca\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.625493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.625519 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-tls\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727024 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.726915 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnkk\" (UniqueName: \"kubernetes.io/projected/df152280-e06e-42df-b5ed-0ee10b3143ab-kube-api-access-6rnkk\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727024 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.726974 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df152280-e06e-42df-b5ed-0ee10b3143ab-metrics-client-ca\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727024 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727020 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-tls\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727355 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727058 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727355 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727103 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727355 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727132 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727355 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727175 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-grpc-tls\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.727355 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727206 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.728065 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.727915 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df152280-e06e-42df-b5ed-0ee10b3143ab-metrics-client-ca\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.733551 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.733037 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-tls\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.733551 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.733486 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-grpc-tls\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.733886 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.733802 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.734169 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.734128 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.734253 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.734175 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.734478 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.734458 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/df152280-e06e-42df-b5ed-0ee10b3143ab-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.736056 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.736017 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnkk\" (UniqueName: \"kubernetes.io/projected/df152280-e06e-42df-b5ed-0ee10b3143ab-kube-api-access-6rnkk\") pod \"thanos-querier-5c5d7468d-dg6nr\" (UID: \"df152280-e06e-42df-b5ed-0ee10b3143ab\") " pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:26.912646 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:26.912548 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:27.084301 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.084266 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5c5d7468d-dg6nr"] Jul 24 15:42:27.374687 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.374647 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" event={"ID":"6c5cb812-a998-476f-9131-3d275a6981ef","Type":"ContainerStarted","Data":"413bf8e05eaed2cfd8d2b70472e4c46c518281f73f694afd4981cebf1c182103"} Jul 24 15:42:27.376987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.376951 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" event={"ID":"f264b9b3-6bc6-4dba-98d9-e29aca740f2d","Type":"ContainerStarted","Data":"50be364a8e9021d7ef236ba3715b2918d84d785a8105e2c02f0ae915f2a78bf0"} Jul 24 15:42:27.377137 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.377011 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" event={"ID":"f264b9b3-6bc6-4dba-98d9-e29aca740f2d","Type":"ContainerStarted","Data":"86d0b433311962f27456ea199751b3cd76b579c8a492869a571674af9d276e2b"} Jul 24 15:42:27.377137 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.377035 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" event={"ID":"f264b9b3-6bc6-4dba-98d9-e29aca740f2d","Type":"ContainerStarted","Data":"6a688db80a4198c6c070228756c884ef49ff9ee76104e9fa1847226b78b4834a"} Jul 24 15:42:27.378641 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.378613 2543 generic.go:356] "Generic (PLEG): container finished" podID="9adb9c66-a5b2-44dc-aa3f-6efc22df5309" containerID="643c2a08c3754fefc77493e29048d595b9bf7e08f522bfad1f0f7e1a98463d04" exitCode=0 Jul 24 15:42:27.378788 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.378693 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbs4d" event={"ID":"9adb9c66-a5b2-44dc-aa3f-6efc22df5309","Type":"ContainerDied","Data":"643c2a08c3754fefc77493e29048d595b9bf7e08f522bfad1f0f7e1a98463d04"} Jul 24 15:42:27.379921 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.379902 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"4953da384c4c547d6b8678dd3bc28f509ffc08cd9ddd6804de0f05f75482433f"} Jul 24 15:42:27.393547 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.393501 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-b8f49f5c6-8cpf8" podStartSLOduration=2.793489732 podStartE2EDuration="4.39348251s" totalImagesPullingTime="1.599992778s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:27.392162348 +0000 UTC m=+78.923749989" watchObservedRunningTime="2026-07-24 15:42:27.39348251 +0000 UTC m=+78.925070115" Jul 24 15:42:27.414550 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.414494 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-dg84v" podStartSLOduration=2.09686513 podStartE2EDuration="4.414477303s" totalImagesPullingTime="2.317612173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:27.414461111 +0000 UTC m=+78.946048726" watchObservedRunningTime="2026-07-24 15:42:27.414477303 +0000 UTC m=+78.946064907" Jul 24 15:42:27.897518 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.897480 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-d4965f97d-dgcj7"] Jul 24 15:42:27.906019 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.905997 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.909550 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.909521 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-flk0dt2btl1n2\"" Jul 24 15:42:27.909851 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.909822 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 24 15:42:27.910104 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.910083 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 24 15:42:27.910279 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.910264 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dmj4g\"" Jul 24 15:42:27.910440 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.910427 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 24 15:42:27.910628 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.910599 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 24 15:42:27.912934 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.912877 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d4965f97d-dgcj7"] Jul 24 15:42:27.939937 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.939901 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.940122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.939947 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-secret-metrics-server-tls\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.940122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.939976 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-client-ca-bundle\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.940122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.939995 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-audit-log\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.940122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.940053 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n72zf\" (UniqueName: \"kubernetes.io/projected/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-kube-api-access-n72zf\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.940122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.940084 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-metrics-server-audit-profiles\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:27.940378 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:27.940159 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-secret-metrics-server-client-certs\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.040900 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.040850 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n72zf\" (UniqueName: \"kubernetes.io/projected/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-kube-api-access-n72zf\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.040900 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.040903 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-metrics-server-audit-profiles\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.040961 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-secret-metrics-server-client-certs\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.040999 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.041028 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-secret-metrics-server-tls\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.041059 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-client-ca-bundle\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041122 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.041088 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-audit-log\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041582 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.041538 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-audit-log\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.041907 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.041857 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.042035 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.042015 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-metrics-server-audit-profiles\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.044342 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.044318 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-secret-metrics-server-tls\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.044456 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.044410 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-secret-metrics-server-client-certs\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.044503 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.044480 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-client-ca-bundle\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.049280 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.049260 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n72zf\" (UniqueName: \"kubernetes.io/projected/ca07aae5-3ae0-4fc3-b01f-7a29bf147094-kube-api-access-n72zf\") pod \"metrics-server-d4965f97d-dgcj7\" (UID: \"ca07aae5-3ae0-4fc3-b01f-7a29bf147094\") " pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.217742 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.217655 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:28.355914 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.355883 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv"] Jul 24 15:42:28.359242 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.359219 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:28.362219 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.362201 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 24 15:42:28.362362 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.362341 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-8h7q7\"" Jul 24 15:42:28.370140 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.370120 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv"] Jul 24 15:42:28.445092 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.445052 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc265e8a-e941-44fe-8fa7-02b49b3ac250-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:28.445817 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.445661 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdm6q\" (UniqueName: \"kubernetes.io/projected/dc265e8a-e941-44fe-8fa7-02b49b3ac250-kube-api-access-pdm6q\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:28.547098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.547054 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdm6q\" (UniqueName: \"kubernetes.io/projected/dc265e8a-e941-44fe-8fa7-02b49b3ac250-kube-api-access-pdm6q\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:28.547302 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.547124 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc265e8a-e941-44fe-8fa7-02b49b3ac250-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:28.547368 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:28.547316 2543 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 24 15:42:28.547420 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:42:28.547384 2543 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc265e8a-e941-44fe-8fa7-02b49b3ac250-monitoring-plugin-cert podName:dc265e8a-e941-44fe-8fa7-02b49b3ac250 nodeName:}" failed. No retries permitted until 2026-07-24 15:42:29.0473604 +0000 UTC m=+80.578948006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/dc265e8a-e941-44fe-8fa7-02b49b3ac250-monitoring-plugin-cert") pod "monitoring-plugin-68b7fc9574-ktwhv" (UID: "dc265e8a-e941-44fe-8fa7-02b49b3ac250") : secret "monitoring-plugin-cert" not found Jul 24 15:42:28.556388 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:28.556351 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdm6q\" (UniqueName: \"kubernetes.io/projected/dc265e8a-e941-44fe-8fa7-02b49b3ac250-kube-api-access-pdm6q\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:29.051546 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.051515 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc265e8a-e941-44fe-8fa7-02b49b3ac250-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:29.054340 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.054317 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/dc265e8a-e941-44fe-8fa7-02b49b3ac250-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-ktwhv\" (UID: \"dc265e8a-e941-44fe-8fa7-02b49b3ac250\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:29.271272 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.271233 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:29.826412 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.826374 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:42:29.864428 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.864396 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:42:29.864638 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.864616 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.866967 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.866949 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 15:42:29.867745 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.867726 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 15:42:29.868032 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.868015 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zljpf\"" Jul 24 15:42:29.868417 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.868402 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 15:42:29.868781 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.868751 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 15:42:29.869009 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.868994 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 15:42:29.869209 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.869196 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 15:42:29.869396 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.869382 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 15:42:29.869678 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.869660 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 15:42:29.869910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.869891 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 15:42:29.870016 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.869920 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 15:42:29.870101 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.870086 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 15:42:29.870432 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.870407 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bvu78enmek85l\"" Jul 24 15:42:29.870520 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.870450 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 15:42:29.883174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.883146 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 15:42:29.883330 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.883291 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 15:42:29.962028 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.961992 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962028 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962041 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962111 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42qh\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-kube-api-access-z42qh\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962159 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962183 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962204 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962239 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-config\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962254 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962288 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962278 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962295 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962321 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962341 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962374 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962462 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962612 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962552 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962613 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962656 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962683 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962708 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:29.962958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:29.962733 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.063660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063626 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063668 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063694 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42qh\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-kube-api-access-z42qh\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063722 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063749 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063781 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063813 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-config\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063836 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063881 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063908 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063930 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063956 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.063985 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064016 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064064 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064092 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064125 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064116 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064936 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064141 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064936 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064165 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064936 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064193 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064936 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064465 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.064936 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.064635 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.065238 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.065211 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.065315 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.065297 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.066407 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.066251 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.067785 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.067997 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.068298 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.068675 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.068880 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.069284 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.071294 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.069484 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.072204 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.071439 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.072204 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.071940 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.072204 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.072091 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-config-out\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.072774 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.072752 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.073655 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.073604 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.073655 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.073620 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-web-config\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.073867 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.073848 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-config\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.075758 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.075736 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42qh\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-kube-api-access-z42qh\") pod \"prometheus-k8s-0\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:30.182920 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:30.182829 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:35.769884 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:35.769838 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf152280_e06e_42df_b5ed_0ee10b3143ab.slice/crio-03b5204bbbb1318b9ccb981df65b52e76eac328d14668fede6996c29ff62e502 WatchSource:0}: Error finding container 03b5204bbbb1318b9ccb981df65b52e76eac328d14668fede6996c29ff62e502: Status 404 returned error can't find the container with id 03b5204bbbb1318b9ccb981df65b52e76eac328d14668fede6996c29ff62e502 Jul 24 15:42:36.418547 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:36.418474 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"03b5204bbbb1318b9ccb981df65b52e76eac328d14668fede6996c29ff62e502"} Jul 24 15:42:36.619989 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:36.619915 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-d4965f97d-dgcj7"] Jul 24 15:42:36.713491 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:36.713466 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca07aae5_3ae0_4fc3_b01f_7a29bf147094.slice/crio-85a47467998b31aa34c164a76316ef62a1dae49b6fd326ef7285266fdd844e6a WatchSource:0}: Error finding container 85a47467998b31aa34c164a76316ef62a1dae49b6fd326ef7285266fdd844e6a: Status 404 returned error can't find the container with id 85a47467998b31aa34c164a76316ef62a1dae49b6fd326ef7285266fdd844e6a Jul 24 15:42:36.849514 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:36.849443 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv"] Jul 24 15:42:36.858754 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:36.858728 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:42:36.866572 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:42:36.865249 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c7271_36df_4471_af94_2c946624e1bf.slice/crio-b11d03320410f1e9753978f25a6d8e70ac3cd19001eb50a1e9300ffcde03980e WatchSource:0}: Error finding container b11d03320410f1e9753978f25a6d8e70ac3cd19001eb50a1e9300ffcde03980e: Status 404 returned error can't find the container with id b11d03320410f1e9753978f25a6d8e70ac3cd19001eb50a1e9300ffcde03980e Jul 24 15:42:37.427788 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.427034 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbs4d" event={"ID":"9adb9c66-a5b2-44dc-aa3f-6efc22df5309","Type":"ContainerStarted","Data":"19d6c65f4634f741bb732db741ec088e37b843820be9a1c285b08a9f0da1daff"} Jul 24 15:42:37.427788 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.427140 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-sbs4d" event={"ID":"9adb9c66-a5b2-44dc-aa3f-6efc22df5309","Type":"ContainerStarted","Data":"48430553c60e4faed80c90c54ebd9dec0d368671c19f9f77cbb0f1f30d05c95b"} Jul 24 15:42:37.428894 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.428844 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-dpctg" event={"ID":"3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7","Type":"ContainerStarted","Data":"85c1b9d5c27ba07b6e4232d2105c3d1a6ac1dc39b6d3f7f79c6d69b5b462e96e"} Jul 24 15:42:37.429046 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.429008 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:37.430637 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.430580 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" event={"ID":"ca07aae5-3ae0-4fc3-b01f-7a29bf147094","Type":"ContainerStarted","Data":"85a47467998b31aa34c164a76316ef62a1dae49b6fd326ef7285266fdd844e6a"} Jul 24 15:42:37.433488 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.433458 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8474c7665c-5k796" event={"ID":"bdb263aa-d586-41c5-b00c-a459330e157d","Type":"ContainerStarted","Data":"6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32"} Jul 24 15:42:37.435721 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.435601 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5" exitCode=0 Jul 24 15:42:37.436048 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.435735 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5"} Jul 24 15:42:37.438237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.438211 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" exitCode=0 Jul 24 15:42:37.438347 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.438293 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} Jul 24 15:42:37.438410 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.438352 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"b11d03320410f1e9753978f25a6d8e70ac3cd19001eb50a1e9300ffcde03980e"} Jul 24 15:42:37.440124 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.440036 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-dpctg" Jul 24 15:42:37.442111 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.442030 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" event={"ID":"dc265e8a-e941-44fe-8fa7-02b49b3ac250","Type":"ContainerStarted","Data":"0a94bf773d8ba327f35ac457fbe3ec19879e8245c08fdc5ce9f8a320029765ef"} Jul 24 15:42:37.454900 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.452059 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-sbs4d" podStartSLOduration=12.494018323 podStartE2EDuration="14.452041984s" totalImagesPullingTime="1.821820019s" totalInitContainerRuntime="136.203642ms" isStatefulPod=true podCreationTimestamp="2026-07-24 15:42:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:37.450271563 +0000 UTC m=+88.981859195" watchObservedRunningTime="2026-07-24 15:42:37.452041984 +0000 UTC m=+88.983629598" Jul 24 15:42:37.497248 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.496950 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-dpctg" podStartSLOduration=1.647373858 podStartE2EDuration="23.496931865s" totalImagesPullingTime="21.849558007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:37.496741758 +0000 UTC m=+89.028329375" watchObservedRunningTime="2026-07-24 15:42:37.496931865 +0000 UTC m=+89.028519477" Jul 24 15:42:37.517291 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:37.517228 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8474c7665c-5k796" podStartSLOduration=1.837806718 podStartE2EDuration="13.517206376s" totalImagesPullingTime="11.679399658s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:37.515618374 +0000 UTC m=+89.047206012" watchObservedRunningTime="2026-07-24 15:42:37.517206376 +0000 UTC m=+89.048793993" Jul 24 15:42:43.482864 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.482653 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} Jul 24 15:42:43.482864 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.482793 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} Jul 24 15:42:43.484499 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.484421 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" event={"ID":"dc265e8a-e941-44fe-8fa7-02b49b3ac250","Type":"ContainerStarted","Data":"eeb438b443ac28633745f1cebb1fa338ecbb21040717579fcf90656aa3b13926"} Jul 24 15:42:43.486755 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.486492 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"e6657431a4c3e334c7a2359f6698e24196b2ab6d0a6a6cf76c6f87a5d0de6a9d"} Jul 24 15:42:43.486755 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.486579 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"74b8d7730feabadc4871555a74d2b695b5ed016b8d42f94062b3932f1130f013"} Jul 24 15:42:43.488732 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.488674 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" event={"ID":"ca07aae5-3ae0-4fc3-b01f-7a29bf147094","Type":"ContainerStarted","Data":"e09f1f65a068e8c5e0fd4db838ff4dc9f1606c2f9e79f5cbfe8aeeebc0eb0dc8"} Jul 24 15:42:43.491906 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.491755 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed"} Jul 24 15:42:43.491906 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.491814 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479"} Jul 24 15:42:43.500987 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.499099 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:43.509662 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.509597 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" Jul 24 15:42:43.515632 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.515589 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-ktwhv" podStartSLOduration=9.326213012 podStartE2EDuration="15.515550943s" totalImagesPullingTime="6.189337931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:43.514912622 +0000 UTC m=+95.046500235" watchObservedRunningTime="2026-07-24 15:42:43.515550943 +0000 UTC m=+95.047138533" Jul 24 15:42:43.644694 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.642231 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" podStartSLOduration=10.316187306 podStartE2EDuration="16.642206271s" totalImagesPullingTime="6.326018965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:43.552301737 +0000 UTC m=+95.083889368" watchObservedRunningTime="2026-07-24 15:42:43.642206271 +0000 UTC m=+95.173793884" Jul 24 15:42:43.645196 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:43.645118 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8474c7665c-5k796"] Jul 24 15:42:44.503906 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.503862 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6"} Jul 24 15:42:44.504445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.503954 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889"} Jul 24 15:42:44.504445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.504000 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c"} Jul 24 15:42:44.510237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.509017 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} Jul 24 15:42:44.510237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.509093 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} Jul 24 15:42:44.510237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.509137 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} Jul 24 15:42:44.510237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.509178 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerStarted","Data":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} Jul 24 15:42:44.514997 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.514453 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"bdc96de90d75e74acf33b98ebf8daf4d4483427b971aa7f8b9f515a1d0773577"} Jul 24 15:42:44.544017 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.543940 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=9.877883315 podStartE2EDuration="15.543919645s" totalImagesPullingTime="5.642361828s" totalInitContainerRuntime="23.674502ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:44.537296502 +0000 UTC m=+96.068884134" watchObservedRunningTime="2026-07-24 15:42:44.543919645 +0000 UTC m=+96.075507264" Jul 24 15:42:44.599946 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:44.599890 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:42:45.183643 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:45.183605 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:42:46.534444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:46.534401 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerStarted","Data":"3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f"} Jul 24 15:42:46.537348 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:46.537321 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"f6cb366e11fe67f5862d8694ac8c65f161263e505864ba26cd6be5227bef380e"} Jul 24 15:42:46.537470 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:46.537391 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"a1e211a8dddd5b1b774e13912e247b08d0cf76ffb607c885456a8582b8d12e78"} Jul 24 15:42:46.537470 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:46.537429 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" event={"ID":"df152280-e06e-42df-b5ed-0ee10b3143ab","Type":"ContainerStarted","Data":"fd2963e2e431fafc8e07a77a17b4f823c7a016c06ef0d08662242fd305da5ddf"} Jul 24 15:42:46.561279 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:46.561228 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=5.282027493 podStartE2EDuration="22.561213333s" totalImagesPullingTime="17.246519024s" totalInitContainerRuntime="32.666816ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:46.558303238 +0000 UTC m=+98.089890852" watchObservedRunningTime="2026-07-24 15:42:46.561213333 +0000 UTC m=+98.092800994" Jul 24 15:42:46.591491 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:46.591421 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" podStartSLOduration=11.826618973 podStartE2EDuration="20.591399774s" totalImagesPullingTime="8.764780801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:42:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:42:46.58676672 +0000 UTC m=+98.118354373" watchObservedRunningTime="2026-07-24 15:42:46.591399774 +0000 UTC m=+98.122987390" Jul 24 15:42:47.541536 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:47.541507 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:48.218046 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:48.218010 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:48.218216 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:48.218074 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:42:48.552248 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:48.552210 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5c5d7468d-dg6nr" Jul 24 15:42:52.344199 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:42:52.344168 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lkzq7" Jul 24 15:43:08.223072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:08.223044 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:43:08.227019 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:08.226997 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-d4965f97d-dgcj7" Jul 24 15:43:08.677679 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:08.677556 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8474c7665c-5k796" podUID="bdb263aa-d586-41c5-b00c-a459330e157d" containerName="console" containerID="cri-o://6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32" gracePeriod=15 Jul 24 15:43:08.959289 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:08.959217 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:43:09.015407 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015372 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rfzk\" (UniqueName: \"kubernetes.io/projected/bdb263aa-d586-41c5-b00c-a459330e157d-kube-api-access-9rfzk\") pod \"bdb263aa-d586-41c5-b00c-a459330e157d\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " Jul 24 15:43:09.015407 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015413 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-oauth-serving-cert\") pod \"bdb263aa-d586-41c5-b00c-a459330e157d\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " Jul 24 15:43:09.015685 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015447 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-serving-cert\") pod \"bdb263aa-d586-41c5-b00c-a459330e157d\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " Jul 24 15:43:09.015685 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015502 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-console-config\") pod \"bdb263aa-d586-41c5-b00c-a459330e157d\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " Jul 24 15:43:09.015685 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015532 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-oauth-config\") pod \"bdb263aa-d586-41c5-b00c-a459330e157d\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " Jul 24 15:43:09.015837 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015759 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-service-ca\") pod \"bdb263aa-d586-41c5-b00c-a459330e157d\" (UID: \"bdb263aa-d586-41c5-b00c-a459330e157d\") " Jul 24 15:43:09.015900 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.015851 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "bdb263aa-d586-41c5-b00c-a459330e157d" (UID: "bdb263aa-d586-41c5-b00c-a459330e157d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:09.016123 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.016098 2543 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-oauth-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:09.016123 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.016104 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-console-config" (OuterVolumeSpecName: "console-config") pod "bdb263aa-d586-41c5-b00c-a459330e157d" (UID: "bdb263aa-d586-41c5-b00c-a459330e157d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:09.016280 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.016122 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-service-ca" (OuterVolumeSpecName: "service-ca") pod "bdb263aa-d586-41c5-b00c-a459330e157d" (UID: "bdb263aa-d586-41c5-b00c-a459330e157d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:09.017952 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.017931 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdb263aa-d586-41c5-b00c-a459330e157d-kube-api-access-9rfzk" (OuterVolumeSpecName: "kube-api-access-9rfzk") pod "bdb263aa-d586-41c5-b00c-a459330e157d" (UID: "bdb263aa-d586-41c5-b00c-a459330e157d"). InnerVolumeSpecName "kube-api-access-9rfzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:09.017952 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.017942 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "bdb263aa-d586-41c5-b00c-a459330e157d" (UID: "bdb263aa-d586-41c5-b00c-a459330e157d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:09.018059 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.018035 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "bdb263aa-d586-41c5-b00c-a459330e157d" (UID: "bdb263aa-d586-41c5-b00c-a459330e157d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:09.117535 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.117502 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rfzk\" (UniqueName: \"kubernetes.io/projected/bdb263aa-d586-41c5-b00c-a459330e157d-kube-api-access-9rfzk\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:09.117535 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.117534 2543 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-serving-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:09.117811 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.117548 2543 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-console-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:09.117811 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.117586 2543 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/bdb263aa-d586-41c5-b00c-a459330e157d-console-oauth-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:09.117811 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.117601 2543 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb263aa-d586-41c5-b00c-a459330e157d-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:09.631817 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.631782 2543 generic.go:356] "Generic (PLEG): container finished" podID="bdb263aa-d586-41c5-b00c-a459330e157d" containerID="6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32" exitCode=0 Jul 24 15:43:09.632277 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.631822 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8474c7665c-5k796" Jul 24 15:43:09.632277 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.631871 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8474c7665c-5k796" event={"ID":"bdb263aa-d586-41c5-b00c-a459330e157d","Type":"ContainerDied","Data":"6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32"} Jul 24 15:43:09.632277 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.631915 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8474c7665c-5k796" event={"ID":"bdb263aa-d586-41c5-b00c-a459330e157d","Type":"ContainerDied","Data":"b87c4047e58502bd5567030c6a4be0f1dac4615c8f41d069288dc94e0c587f26"} Jul 24 15:43:09.632277 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.631934 2543 scope.go:117] "RemoveContainer" containerID="6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32" Jul 24 15:43:09.641306 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.641290 2543 scope.go:117] "RemoveContainer" containerID="6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32" Jul 24 15:43:09.641590 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:09.641548 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32\": container with ID starting with 6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32 not found: ID does not exist" containerID="6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32" Jul 24 15:43:09.641663 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.641600 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32"} err="failed to get container status \"6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32\": rpc error: code = NotFound desc = could not find container \"6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32\": container with ID starting with 6de3c4cfa0b2c8bd87bc48f7ac90e3875cb622c444354f5fdd7199c1732dfa32 not found: ID does not exist" Jul 24 15:43:09.647788 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.647768 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8474c7665c-5k796"] Jul 24 15:43:09.651876 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:09.651858 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8474c7665c-5k796"] Jul 24 15:43:11.042677 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:11.042648 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb263aa-d586-41c5-b00c-a459330e157d" path="/var/lib/kubelet/pods/bdb263aa-d586-41c5-b00c-a459330e157d/volumes" Jul 24 15:43:30.183878 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:30.183841 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:30.242574 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:30.242540 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:30.730789 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:30.730763 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:43.840580 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.840539 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:43:43.841167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.840940 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy" containerID="cri-o://386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889" gracePeriod=120 Jul 24 15:43:43.841167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.840940 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="alertmanager" containerID="cri-o://99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479" gracePeriod=120 Jul 24 15:43:43.841167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.840985 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-web" containerID="cri-o://c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c" gracePeriod=120 Jul 24 15:43:43.841167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.841040 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-metric" containerID="cri-o://8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6" gracePeriod=120 Jul 24 15:43:43.841167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.841004 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="prom-label-proxy" containerID="cri-o://3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f" gracePeriod=120 Jul 24 15:43:43.841167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:43.841004 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="config-reloader" containerID="cri-o://fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed" gracePeriod=120 Jul 24 15:43:44.774250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774216 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f" exitCode=0 Jul 24 15:43:44.774250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774240 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6" exitCode=0 Jul 24 15:43:44.774250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774247 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889" exitCode=0 Jul 24 15:43:44.774250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774252 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed" exitCode=0 Jul 24 15:43:44.774250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774257 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479" exitCode=0 Jul 24 15:43:44.774534 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774288 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f"} Jul 24 15:43:44.774534 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774351 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6"} Jul 24 15:43:44.774534 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774377 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889"} Jul 24 15:43:44.774534 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774402 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed"} Jul 24 15:43:44.774534 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:44.774425 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479"} Jul 24 15:43:45.078541 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.078520 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.126753 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126723 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-cluster-tls-config\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.126910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126769 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-tls-assets\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.126910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126806 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-metrics-client-ca\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.126910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126831 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.126910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126853 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-volume\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.126910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126887 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126931 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qx9cf\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-kube-api-access-qx9cf\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126965 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.126992 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-main-db\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127025 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-web-config\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127059 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-web\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127086 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-out\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127174 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127111 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle\") pod \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\" (UID: \"43f33209-9f25-4418-9c7b-e7eaf9adaa5c\") " Jul 24 15:43:45.127489 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127183 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:45.127489 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127321 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:45.127616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127523 2543 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-main-db\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.127616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127544 2543 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-metrics-client-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.127737 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.127713 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:45.129681 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.129638 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:45.130984 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.130942 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.131208 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.131179 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.131457 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.131426 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-kube-api-access-qx9cf" (OuterVolumeSpecName: "kube-api-access-qx9cf") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "kube-api-access-qx9cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:45.131736 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.131491 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-volume" (OuterVolumeSpecName: "config-volume") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.132576 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.132524 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-out" (OuterVolumeSpecName: "config-out") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:45.133193 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.133158 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.133298 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.133274 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.136809 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.136784 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.142737 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.142641 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-web-config" (OuterVolumeSpecName: "web-config") pod "43f33209-9f25-4418-9c7b-e7eaf9adaa5c" (UID: "43f33209-9f25-4418-9c7b-e7eaf9adaa5c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:45.228208 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228178 2543 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-cluster-tls-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228208 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228201 2543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-tls-assets\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228208 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228210 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-main-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228218 2543 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228226 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228235 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx9cf\" (UniqueName: \"kubernetes.io/projected/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-kube-api-access-qx9cf\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228243 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228252 2543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-web-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228260 2543 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228269 2543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-config-out\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.228409 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.228276 2543 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/43f33209-9f25-4418-9c7b-e7eaf9adaa5c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:45.781183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.781150 2543 generic.go:356] "Generic (PLEG): container finished" podID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerID="c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c" exitCode=0 Jul 24 15:43:45.781183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.781192 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c"} Jul 24 15:43:45.781449 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.781241 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.781449 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.781299 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"43f33209-9f25-4418-9c7b-e7eaf9adaa5c","Type":"ContainerDied","Data":"4953da384c4c547d6b8678dd3bc28f509ffc08cd9ddd6804de0f05f75482433f"} Jul 24 15:43:45.781449 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.781349 2543 scope.go:117] "RemoveContainer" containerID="3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f" Jul 24 15:43:45.789620 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.789602 2543 scope.go:117] "RemoveContainer" containerID="8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6" Jul 24 15:43:45.796725 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.796683 2543 scope.go:117] "RemoveContainer" containerID="386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889" Jul 24 15:43:45.803970 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.803952 2543 scope.go:117] "RemoveContainer" containerID="c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c" Jul 24 15:43:45.804646 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.804554 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:43:45.809880 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.809860 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:43:45.811417 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.811403 2543 scope.go:117] "RemoveContainer" containerID="fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed" Jul 24 15:43:45.818065 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.818046 2543 scope.go:117] "RemoveContainer" containerID="99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479" Jul 24 15:43:45.824776 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.824762 2543 scope.go:117] "RemoveContainer" containerID="74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5" Jul 24 15:43:45.831354 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.831339 2543 scope.go:117] "RemoveContainer" containerID="3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f" Jul 24 15:43:45.831612 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.831593 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f\": container with ID starting with 3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f not found: ID does not exist" containerID="3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f" Jul 24 15:43:45.831675 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.831627 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f"} err="failed to get container status \"3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f\": rpc error: code = NotFound desc = could not find container \"3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f\": container with ID starting with 3c561eb9db2cf079e9f5410851ba65219e11e458591ab19c65b6d127946ca28f not found: ID does not exist" Jul 24 15:43:45.831675 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.831653 2543 scope.go:117] "RemoveContainer" containerID="8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6" Jul 24 15:43:45.831925 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.831907 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6\": container with ID starting with 8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6 not found: ID does not exist" containerID="8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6" Jul 24 15:43:45.831964 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.831932 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6"} err="failed to get container status \"8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6\": rpc error: code = NotFound desc = could not find container \"8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6\": container with ID starting with 8a21c855b491a10a6d07d3fdab65632436770a1b2707f90f0f956715b8c6a6d6 not found: ID does not exist" Jul 24 15:43:45.831964 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.831948 2543 scope.go:117] "RemoveContainer" containerID="386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889" Jul 24 15:43:45.832168 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.832150 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889\": container with ID starting with 386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889 not found: ID does not exist" containerID="386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889" Jul 24 15:43:45.832236 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832176 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889"} err="failed to get container status \"386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889\": rpc error: code = NotFound desc = could not find container \"386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889\": container with ID starting with 386df071e4862cd2c0e0a421ae83722afb7431977cb4370ae604b7dfc3f14889 not found: ID does not exist" Jul 24 15:43:45.832236 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832195 2543 scope.go:117] "RemoveContainer" containerID="c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c" Jul 24 15:43:45.832443 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.832427 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c\": container with ID starting with c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c not found: ID does not exist" containerID="c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c" Jul 24 15:43:45.832482 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832448 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c"} err="failed to get container status \"c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c\": rpc error: code = NotFound desc = could not find container \"c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c\": container with ID starting with c4d148b40c633bd4b1913e3ad4bb39c4a30323eeeb0487015741fa11774f692c not found: ID does not exist" Jul 24 15:43:45.832482 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832461 2543 scope.go:117] "RemoveContainer" containerID="fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed" Jul 24 15:43:45.832682 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.832666 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed\": container with ID starting with fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed not found: ID does not exist" containerID="fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed" Jul 24 15:43:45.832750 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832689 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed"} err="failed to get container status \"fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed\": rpc error: code = NotFound desc = could not find container \"fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed\": container with ID starting with fee5c6044e227e8ff0a9911fa240dc2fd20ed3e13d1233c09032e85f48dc3aed not found: ID does not exist" Jul 24 15:43:45.832750 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832707 2543 scope.go:117] "RemoveContainer" containerID="99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479" Jul 24 15:43:45.832922 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.832907 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479\": container with ID starting with 99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479 not found: ID does not exist" containerID="99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479" Jul 24 15:43:45.832961 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832928 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479"} err="failed to get container status \"99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479\": rpc error: code = NotFound desc = could not find container \"99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479\": container with ID starting with 99e1dd4f23e9c393383db92b5fc04ae159a400861b3c0a882263b5835c5b0479 not found: ID does not exist" Jul 24 15:43:45.832961 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.832941 2543 scope.go:117] "RemoveContainer" containerID="74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5" Jul 24 15:43:45.833149 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:45.833130 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5\": container with ID starting with 74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5 not found: ID does not exist" containerID="74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5" Jul 24 15:43:45.833199 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.833153 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5"} err="failed to get container status \"74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5\": rpc error: code = NotFound desc = could not find container \"74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5\": container with ID starting with 74004acebb2b3b5a8d4362bf0ad9c5f2e35d1f14cae0d5fd4b6a98306fbc15f5 not found: ID does not exist" Jul 24 15:43:45.871389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871362 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:43:45.871673 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871660 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="config-reloader" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871675 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="config-reloader" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871684 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="init-config-reloader" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871690 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="init-config-reloader" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871705 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="prom-label-proxy" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871710 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="prom-label-proxy" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871720 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871724 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy" Jul 24 15:43:45.871727 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871730 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb263aa-d586-41c5-b00c-a459330e157d" containerName="console" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871736 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb263aa-d586-41c5-b00c-a459330e157d" containerName="console" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871741 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-metric" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871746 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-metric" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871754 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="alertmanager" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871759 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="alertmanager" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871768 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-web" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871773 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-web" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871820 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-metric" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871829 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871836 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="alertmanager" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871842 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb263aa-d586-41c5-b00c-a459330e157d" containerName="console" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871848 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="config-reloader" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871853 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="kube-rbac-proxy-web" Jul 24 15:43:45.872021 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.871860 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" containerName="prom-label-proxy" Jul 24 15:43:45.875502 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.875480 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.878079 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878050 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 24 15:43:45.878170 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878081 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 24 15:43:45.878170 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878050 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 24 15:43:45.878170 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878153 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 24 15:43:45.878369 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878355 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 24 15:43:45.878429 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878408 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 24 15:43:45.878482 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878443 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 24 15:43:45.878536 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878501 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zcdjr\"" Jul 24 15:43:45.878628 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.878613 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 24 15:43:45.884328 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.884306 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 24 15:43:45.891860 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.891842 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:43:45.933783 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933749 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06d847ab-a11e-47df-92c1-b926627962ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.933965 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933791 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d847ab-a11e-47df-92c1-b926627962ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.933965 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933816 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.933965 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933866 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06d847ab-a11e-47df-92c1-b926627962ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.933965 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933910 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933965 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9gx9\" (UniqueName: \"kubernetes.io/projected/06d847ab-a11e-47df-92c1-b926627962ab-kube-api-access-j9gx9\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.933999 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.934017 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.934064 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.934131 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-web-config\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934167 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.934151 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06d847ab-a11e-47df-92c1-b926627962ab-config-out\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934462 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.934169 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:45.934462 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:45.934205 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d847ab-a11e-47df-92c1-b926627962ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035016 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.034929 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035016 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.034960 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06d847ab-a11e-47df-92c1-b926627962ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035016 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.034978 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035075 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9gx9\" (UniqueName: \"kubernetes.io/projected/06d847ab-a11e-47df-92c1-b926627962ab-kube-api-access-j9gx9\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035094 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035111 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035131 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035157 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-web-config\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035175 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06d847ab-a11e-47df-92c1-b926627962ab-config-out\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035197 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035228 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d847ab-a11e-47df-92c1-b926627962ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035275 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035257 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06d847ab-a11e-47df-92c1-b926627962ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035734 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035286 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d847ab-a11e-47df-92c1-b926627962ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.035734 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.035464 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/06d847ab-a11e-47df-92c1-b926627962ab-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.036205 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.036177 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/06d847ab-a11e-47df-92c1-b926627962ab-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.036759 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.036670 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06d847ab-a11e-47df-92c1-b926627962ab-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.038663 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.038451 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-web-config\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.038771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.038754 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/06d847ab-a11e-47df-92c1-b926627962ab-config-out\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.038843 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.038775 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.038897 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.038850 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-config-volume\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.039100 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.039074 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.039179 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.039157 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.039236 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.039200 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.039956 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.039939 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/06d847ab-a11e-47df-92c1-b926627962ab-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.040110 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.040095 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/06d847ab-a11e-47df-92c1-b926627962ab-tls-assets\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.044435 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.044415 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9gx9\" (UniqueName: \"kubernetes.io/projected/06d847ab-a11e-47df-92c1-b926627962ab-kube-api-access-j9gx9\") pod \"alertmanager-main-0\" (UID: \"06d847ab-a11e-47df-92c1-b926627962ab\") " pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.185856 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.185832 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 24 15:43:46.317858 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.317831 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 24 15:43:46.319703 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:43:46.319676 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d847ab_a11e_47df_92c1_b926627962ab.slice/crio-37920813144e243773ce083b2074714ba3d5a7736e267a64b06d7be69db7ee96 WatchSource:0}: Error finding container 37920813144e243773ce083b2074714ba3d5a7736e267a64b06d7be69db7ee96: Status 404 returned error can't find the container with id 37920813144e243773ce083b2074714ba3d5a7736e267a64b06d7be69db7ee96 Jul 24 15:43:46.786461 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.786430 2543 generic.go:356] "Generic (PLEG): container finished" podID="06d847ab-a11e-47df-92c1-b926627962ab" containerID="864bfd66b1972ad495e4d835b4e0c34f3338fa53600b9b7556e8795d26a27097" exitCode=0 Jul 24 15:43:46.786619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.786519 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerDied","Data":"864bfd66b1972ad495e4d835b4e0c34f3338fa53600b9b7556e8795d26a27097"} Jul 24 15:43:46.786619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:46.786604 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"37920813144e243773ce083b2074714ba3d5a7736e267a64b06d7be69db7ee96"} Jul 24 15:43:47.044866 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.044794 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43f33209-9f25-4418-9c7b-e7eaf9adaa5c" path="/var/lib/kubelet/pods/43f33209-9f25-4418-9c7b-e7eaf9adaa5c/volumes" Jul 24 15:43:47.797025 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.796991 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"eb21cc4387a68c460c97e7aea12c3040be267c3eee145a91454480f23f9f5dc3"} Jul 24 15:43:47.797364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.797072 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"8e5155e959c92819e1c1f595cf257a52920bab732ef3b6918f0e73e56e06725c"} Jul 24 15:43:47.797364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.797114 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"3432ddae617dcb533d73811fd346a5e3ebf53b5225da08bb2889ec20172e41ce"} Jul 24 15:43:47.797364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.797153 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"8991d22f304b0c27677d2dc10142df0bb87c1019d149af9ed89d6bda6817ae54"} Jul 24 15:43:47.797364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.797189 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"d9f29621732ce1056ac6e639da98a32795fa611a2c84cd79caa558c22a3cfb47"} Jul 24 15:43:47.797364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.797224 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"06d847ab-a11e-47df-92c1-b926627962ab","Type":"ContainerStarted","Data":"10c93d2fb6d6635afcb3ae8ee5882e6951e34ecb4f29e33ccb64e9b44f02a0dd"} Jul 24 15:43:47.827969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.827912 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.811509848 podStartE2EDuration="2.827892036s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.382188ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:47.825180968 +0000 UTC m=+159.356768615" watchObservedRunningTime="2026-07-24 15:43:47.827892036 +0000 UTC m=+159.359479650" Jul 24 15:43:47.853665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.853639 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn"] Jul 24 15:43:47.856660 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.856180 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.859289 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.859262 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-pwbhc\"" Jul 24 15:43:47.859421 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.859292 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 24 15:43:47.859908 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.859742 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 24 15:43:47.859908 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.859784 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 24 15:43:47.859908 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.859815 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 24 15:43:47.860135 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.860019 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 24 15:43:47.866722 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.866700 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 24 15:43:47.870675 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.870657 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn"] Jul 24 15:43:47.952424 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952394 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smv6v\" (UniqueName: \"kubernetes.io/projected/120bd707-413d-430c-ab1a-94dab7beeb21-kube-api-access-smv6v\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952435 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952460 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-secret-telemeter-client\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952579 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-serving-certs-ca-bundle\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952607 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-metrics-client-ca\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952752 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952672 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-federate-client-tls\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952752 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952703 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-telemeter-client-tls\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:47.952752 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:47.952741 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.053939 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.053854 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smv6v\" (UniqueName: \"kubernetes.io/projected/120bd707-413d-430c-ab1a-94dab7beeb21-kube-api-access-smv6v\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.053939 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.053919 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054168 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054002 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-secret-telemeter-client\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054168 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054081 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-serving-certs-ca-bundle\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054168 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054109 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-metrics-client-ca\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054323 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054257 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-federate-client-tls\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054323 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054296 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-telemeter-client-tls\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054420 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054361 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.054810 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054787 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-metrics-client-ca\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.055086 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054816 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-telemeter-trusted-ca-bundle\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.055086 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.054928 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/120bd707-413d-430c-ab1a-94dab7beeb21-serving-certs-ca-bundle\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.056974 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.056952 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-secret-telemeter-client\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.057052 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.057007 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.057052 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.057035 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-federate-client-tls\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.057496 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.057473 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/120bd707-413d-430c-ab1a-94dab7beeb21-telemeter-client-tls\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.064298 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.064274 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smv6v\" (UniqueName: \"kubernetes.io/projected/120bd707-413d-430c-ab1a-94dab7beeb21-kube-api-access-smv6v\") pod \"telemeter-client-c7f9fccb9-n4qzn\" (UID: \"120bd707-413d-430c-ab1a-94dab7beeb21\") " pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.162944 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.162906 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:43:48.163216 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.163178 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="prometheus" containerID="cri-o://88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" gracePeriod=600 Jul 24 15:43:48.163379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.163208 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy" containerID="cri-o://292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" gracePeriod=600 Jul 24 15:43:48.163379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.163243 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-web" containerID="cri-o://7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" gracePeriod=600 Jul 24 15:43:48.163379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.163256 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="config-reloader" containerID="cri-o://163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" gracePeriod=600 Jul 24 15:43:48.163379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.163260 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" gracePeriod=600 Jul 24 15:43:48.163379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.163217 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="thanos-sidecar" containerID="cri-o://86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" gracePeriod=600 Jul 24 15:43:48.174888 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.172868 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" Jul 24 15:43:48.347837 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.347811 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn"] Jul 24 15:43:48.349334 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:43:48.349307 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod120bd707_413d_430c_ab1a_94dab7beeb21.slice/crio-cbeafea1a59eea5ce5929acd65d736a34b3f24f27d7b11237b9288147fb8b361 WatchSource:0}: Error finding container cbeafea1a59eea5ce5929acd65d736a34b3f24f27d7b11237b9288147fb8b361: Status 404 returned error can't find the container with id cbeafea1a59eea5ce5929acd65d736a34b3f24f27d7b11237b9288147fb8b361 Jul 24 15:43:48.423744 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.423719 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457552 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-config-out\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457616 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-tls\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457647 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-kubelet-serving-ca-bundle\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457695 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-1\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457722 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-serving-certs-ca-bundle\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457766 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-web-config\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457787 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-grpc-tls\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457811 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-thanos-prometheus-http-client-file\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457836 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457861 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-metrics-client-certs\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457887 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-trusted-ca-bundle\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457915 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-config\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457945 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z42qh\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-kube-api-access-z42qh\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457968 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-metrics-client-ca\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.457999 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-db\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.458035 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-tls-assets\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.458061 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-0\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.461603 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.458107 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.458147 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-2\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.458189 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-kube-rbac-proxy\") pod \"0b0c7271-36df-4471-af94-2c946624e1bf\" (UID: \"0b0c7271-36df-4471-af94-2c946624e1bf\") " Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.459103 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.460309 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.460645 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.460859 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.462544 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.461339 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.462888 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.462674 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.462936 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.462917 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.463187 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.463148 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:43:48.464145 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.464118 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.465978 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.465943 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-config" (OuterVolumeSpecName: "config") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.467692 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.467656 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.468518 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.468741 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.468839 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.468940 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.469049 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.469062 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-kube-api-access-z42qh" (OuterVolumeSpecName: "kube-api-access-z42qh") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "kube-api-access-z42qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:43:48.470689 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.469358 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.471402 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.471204 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-config-out" (OuterVolumeSpecName: "config-out") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:43:48.481059 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.481038 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-web-config" (OuterVolumeSpecName: "web-config") pod "0b0c7271-36df-4471-af94-2c946624e1bf" (UID: "0b0c7271-36df-4471-af94-2c946624e1bf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 15:43:48.558997 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.558971 2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.558997 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.558998 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559008 2543 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-kube-rbac-proxy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559016 2543 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-config-out\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559026 2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559036 2543 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559045 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559053 2543 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559061 2543 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-web-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559068 2543 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-grpc-tls\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559076 2543 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-thanos-prometheus-http-client-file\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559083 2543 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559092 2543 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-secret-metrics-client-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559101 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559109 2543 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/0b0c7271-36df-4471-af94-2c946624e1bf-config\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559118 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42qh\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-kube-api-access-z42qh\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559127 2543 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-configmap-metrics-client-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559135 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-db\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559142 2543 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0b0c7271-36df-4471-af94-2c946624e1bf-tls-assets\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.559183 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.559150 2543 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0b0c7271-36df-4471-af94-2c946624e1bf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:43:48.806925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806897 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" exitCode=0 Jul 24 15:43:48.806925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806923 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" exitCode=0 Jul 24 15:43:48.806925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806932 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" exitCode=0 Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806941 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" exitCode=0 Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806949 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" exitCode=0 Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806957 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c7271-36df-4471-af94-2c946624e1bf" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" exitCode=0 Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806988 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.806982 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807136 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807183 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807202 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807225 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807267 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807306 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} Jul 24 15:43:48.807460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.807345 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0b0c7271-36df-4471-af94-2c946624e1bf","Type":"ContainerDied","Data":"b11d03320410f1e9753978f25a6d8e70ac3cd19001eb50a1e9300ffcde03980e"} Jul 24 15:43:48.815356 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.815330 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" event={"ID":"120bd707-413d-430c-ab1a-94dab7beeb21","Type":"ContainerStarted","Data":"cbeafea1a59eea5ce5929acd65d736a34b3f24f27d7b11237b9288147fb8b361"} Jul 24 15:43:48.823402 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.823370 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.831718 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.831700 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.836977 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.836952 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:43:48.839774 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.839751 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.841509 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.841492 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:43:48.846451 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.846437 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.853115 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.853091 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.860546 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.860526 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.868391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868370 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:43:48.868726 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868709 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.868794 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868779 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="prometheus" Jul 24 15:43:48.868832 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868799 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="prometheus" Jul 24 15:43:48.868832 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868809 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy" Jul 24 15:43:48.868832 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868815 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy" Jul 24 15:43:48.868832 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868830 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-thanos" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868835 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-thanos" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868843 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="config-reloader" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868848 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="config-reloader" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868858 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="thanos-sidecar" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868863 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="thanos-sidecar" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868876 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="init-config-reloader" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868881 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="init-config-reloader" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868892 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-web" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868899 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-web" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868960 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-web" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868972 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy-thanos" Jul 24 15:43:48.868973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868979 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="kube-rbac-proxy" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868987 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="thanos-sidecar" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868992 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="config-reloader" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.868998 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" containerName="prometheus" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.869002 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.869035 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} err="failed to get container status \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.869061 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.869340 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.869366 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} err="failed to get container status \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" Jul 24 15:43:48.869444 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.869381 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.869881 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.869640 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.869881 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.869669 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} err="failed to get container status \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" Jul 24 15:43:48.869881 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.869690 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.870154 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.870082 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.870154 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.870112 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} err="failed to get container status \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" Jul 24 15:43:48.870154 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.870130 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.870738 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.870385 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.870738 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.870409 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} err="failed to get container status \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" Jul 24 15:43:48.870738 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.870428 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.870738 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.870695 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.870738 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.870719 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} err="failed to get container status \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" Jul 24 15:43:48.870738 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.870740 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.870999 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:43:48.870982 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.871040 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871004 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} err="failed to get container status \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" Jul 24 15:43:48.871040 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871019 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.871271 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871256 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} err="failed to get container status \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" Jul 24 15:43:48.871308 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871272 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.871488 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871472 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} err="failed to get container status \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" Jul 24 15:43:48.871526 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871491 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.871737 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871719 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} err="failed to get container status \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" Jul 24 15:43:48.871781 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871740 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.872002 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.871985 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} err="failed to get container status \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" Jul 24 15:43:48.872041 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872004 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.872217 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872199 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} err="failed to get container status \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" Jul 24 15:43:48.872257 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872219 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.872419 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872404 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} err="failed to get container status \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" Jul 24 15:43:48.872456 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872420 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.872677 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872657 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} err="failed to get container status \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" Jul 24 15:43:48.872732 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872680 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.872928 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872911 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} err="failed to get container status \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" Jul 24 15:43:48.872995 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.872928 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.873084 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873068 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.873138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873113 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} err="failed to get container status \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" Jul 24 15:43:48.873138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873127 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.873472 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873338 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} err="failed to get container status \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" Jul 24 15:43:48.873472 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873373 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.873716 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873674 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} err="failed to get container status \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" Jul 24 15:43:48.873716 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873704 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.873978 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873956 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} err="failed to get container status \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" Jul 24 15:43:48.874035 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.873979 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.874239 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.874221 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} err="failed to get container status \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" Jul 24 15:43:48.874307 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.874256 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.874532 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.874494 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} err="failed to get container status \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" Jul 24 15:43:48.874638 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.874532 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.874824 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.874805 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} err="failed to get container status \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" Jul 24 15:43:48.874912 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.874828 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.875126 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875106 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} err="failed to get container status \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" Jul 24 15:43:48.875190 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875126 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.875301 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875282 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 24 15:43:48.875364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875304 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 24 15:43:48.875364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875347 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} err="failed to get container status \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" Jul 24 15:43:48.875460 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875370 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.875684 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875663 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 24 15:43:48.875785 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875663 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 24 15:43:48.875785 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875718 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} err="failed to get container status \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" Jul 24 15:43:48.875785 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875741 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.875949 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875844 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 24 15:43:48.875949 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.875913 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 24 15:43:48.876202 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876184 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 24 15:43:48.876326 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876297 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} err="failed to get container status \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" Jul 24 15:43:48.876326 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876319 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.876586 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876449 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bvu78enmek85l\"" Jul 24 15:43:48.876586 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876482 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 24 15:43:48.876586 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876499 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 24 15:43:48.876586 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876516 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 24 15:43:48.876586 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876456 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 24 15:43:48.876871 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876649 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} err="failed to get container status \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" Jul 24 15:43:48.876871 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876670 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.876971 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.876903 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 24 15:43:48.877101 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877071 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} err="failed to get container status \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" Jul 24 15:43:48.877204 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877101 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.877354 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877338 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-zljpf\"" Jul 24 15:43:48.877430 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877396 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} err="failed to get container status \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" Jul 24 15:43:48.877430 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877418 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.877716 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877691 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} err="failed to get container status \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" Jul 24 15:43:48.877716 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877715 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.877973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877950 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} err="failed to get container status \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" Jul 24 15:43:48.877973 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.877974 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.878292 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.878271 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} err="failed to get container status \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" Jul 24 15:43:48.878369 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.878292 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.878598 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.878536 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} err="failed to get container status \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" Jul 24 15:43:48.878598 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.878587 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.878847 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.878823 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} err="failed to get container status \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" Jul 24 15:43:48.878847 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.878847 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.879042 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879024 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 24 15:43:48.879162 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879145 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} err="failed to get container status \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" Jul 24 15:43:48.879202 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879164 2543 scope.go:117] "RemoveContainer" containerID="a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149" Jul 24 15:43:48.879415 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879397 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149"} err="failed to get container status \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": rpc error: code = NotFound desc = could not find container \"a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149\": container with ID starting with a3158d44124c69794bc9a1df36eb096841939f00a7fd3191f7089257a03c2149 not found: ID does not exist" Jul 24 15:43:48.879461 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879416 2543 scope.go:117] "RemoveContainer" containerID="292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27" Jul 24 15:43:48.879681 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879664 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27"} err="failed to get container status \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": rpc error: code = NotFound desc = could not find container \"292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27\": container with ID starting with 292cbbf1f2fb285c99120f49dc905ab32bbd6cfe0d35a2d88d69e68baa468b27 not found: ID does not exist" Jul 24 15:43:48.879726 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879682 2543 scope.go:117] "RemoveContainer" containerID="7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5" Jul 24 15:43:48.880012 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.879990 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5"} err="failed to get container status \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": rpc error: code = NotFound desc = could not find container \"7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5\": container with ID starting with 7f056b99cd8207dbc0acaab6309c4f71551755e5e954dfc47fa05d028db8eea5 not found: ID does not exist" Jul 24 15:43:48.880085 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880014 2543 scope.go:117] "RemoveContainer" containerID="86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7" Jul 24 15:43:48.880335 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880307 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7"} err="failed to get container status \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": rpc error: code = NotFound desc = could not find container \"86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7\": container with ID starting with 86854082ab276e8babd8452970f7b99f64860c4d57c2d1d46482e0d9394d43c7 not found: ID does not exist" Jul 24 15:43:48.880463 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880338 2543 scope.go:117] "RemoveContainer" containerID="163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e" Jul 24 15:43:48.880684 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880662 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e"} err="failed to get container status \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": rpc error: code = NotFound desc = could not find container \"163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e\": container with ID starting with 163fc2df5fbee2ac4418c11fce9f64827b42038b815ddabb2daee128553e533e not found: ID does not exist" Jul 24 15:43:48.880765 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880687 2543 scope.go:117] "RemoveContainer" containerID="88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3" Jul 24 15:43:48.880947 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880926 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3"} err="failed to get container status \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": rpc error: code = NotFound desc = could not find container \"88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3\": container with ID starting with 88c58501d37e62ba368ad4041e4428f1450cc2521c7ffc851c2f76d1f730d8a3 not found: ID does not exist" Jul 24 15:43:48.881058 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.880947 2543 scope.go:117] "RemoveContainer" containerID="509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb" Jul 24 15:43:48.881226 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.881209 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 24 15:43:48.881296 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.881225 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb"} err="failed to get container status \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": rpc error: code = NotFound desc = could not find container \"509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb\": container with ID starting with 509b1414769f8050709ad6327091439ec338c6077220aaf9286cddea45195edb not found: ID does not exist" Jul 24 15:43:48.887645 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.887626 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:43:48.964087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964057 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964084 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964110 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964140 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964230 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964261 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964289 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7851499f-f916-47c4-8b6c-fe1e9a120c0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964314 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964341 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964370 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5zjl\" (UniqueName: \"kubernetes.io/projected/7851499f-f916-47c4-8b6c-fe1e9a120c0b-kube-api-access-z5zjl\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964407 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964423 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-config\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964438 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964462 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964581 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7851499f-f916-47c4-8b6c-fe1e9a120c0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964619 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964613 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964641 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964672 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964702 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:48.964969 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:48.964719 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.044953 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.044922 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0c7271-36df-4471-af94-2c946624e1bf" path="/var/lib/kubelet/pods/0b0c7271-36df-4471-af94-2c946624e1bf/volumes" Jul 24 15:43:49.065532 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065511 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065723 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065550 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5zjl\" (UniqueName: \"kubernetes.io/projected/7851499f-f916-47c4-8b6c-fe1e9a120c0b-kube-api-access-z5zjl\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065787 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065751 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065842 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065803 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-config\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065842 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065820 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065940 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065849 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065996 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065941 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7851499f-f916-47c4-8b6c-fe1e9a120c0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065996 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065962 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.065996 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.065969 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066026 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066063 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066091 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066115 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066144 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066168 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066197 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066229 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066270 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066302 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066614 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066331 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7851499f-f916-47c4-8b6c-fe1e9a120c0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066614 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066354 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066614 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066505 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.066882 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.066862 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.069639 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.067192 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.069639 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.067802 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.069639 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.068405 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.069639 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.069101 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.069639 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.069405 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.069929 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.069893 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-config\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.070420 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.070278 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.071453 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.071428 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.072051 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.072029 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-web-config\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.072777 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.072754 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7851499f-f916-47c4-8b6c-fe1e9a120c0b-config-out\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.073816 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.073659 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.073816 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.073761 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.073816 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.073775 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7851499f-f916-47c4-8b6c-fe1e9a120c0b-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.074029 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.073855 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7851499f-f916-47c4-8b6c-fe1e9a120c0b-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.074708 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.074619 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.075240 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.075221 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/7851499f-f916-47c4-8b6c-fe1e9a120c0b-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.075505 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.075482 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5zjl\" (UniqueName: \"kubernetes.io/projected/7851499f-f916-47c4-8b6c-fe1e9a120c0b-kube-api-access-z5zjl\") pod \"prometheus-k8s-0\" (UID: \"7851499f-f916-47c4-8b6c-fe1e9a120c0b\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.183817 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.183787 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:43:49.342949 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.342334 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 24 15:43:49.345363 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:43:49.345330 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7851499f_f916_47c4_8b6c_fe1e9a120c0b.slice/crio-4efed09b3c52fe8a16133824eea491369258274571000a8ec82351d6a2c3ac31 WatchSource:0}: Error finding container 4efed09b3c52fe8a16133824eea491369258274571000a8ec82351d6a2c3ac31: Status 404 returned error can't find the container with id 4efed09b3c52fe8a16133824eea491369258274571000a8ec82351d6a2c3ac31 Jul 24 15:43:49.824085 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.824049 2543 generic.go:356] "Generic (PLEG): container finished" podID="7851499f-f916-47c4-8b6c-fe1e9a120c0b" containerID="6785fe891750f8c72ee44574aab7bfd0936694091c16afeceaf60275c65cb1cb" exitCode=0 Jul 24 15:43:49.824542 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.824125 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerDied","Data":"6785fe891750f8c72ee44574aab7bfd0936694091c16afeceaf60275c65cb1cb"} Jul 24 15:43:49.824542 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:49.824192 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"4efed09b3c52fe8a16133824eea491369258274571000a8ec82351d6a2c3ac31"} Jul 24 15:43:50.830907 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.830870 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" event={"ID":"120bd707-413d-430c-ab1a-94dab7beeb21","Type":"ContainerStarted","Data":"04730cab758f65ded118443726d21163d1c7becc02ac65888b3dd13063c464fb"} Jul 24 15:43:50.831323 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.830935 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" event={"ID":"120bd707-413d-430c-ab1a-94dab7beeb21","Type":"ContainerStarted","Data":"9982b843d2f4ec75342aa59feb9846e30e1f1d642260e54076d4b9736326e08e"} Jul 24 15:43:50.831323 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.830967 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" event={"ID":"120bd707-413d-430c-ab1a-94dab7beeb21","Type":"ContainerStarted","Data":"87ccaec081929c7459ac24129710d016b49ea619e5854f2ce26941f893163b13"} Jul 24 15:43:50.833884 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.833856 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"ea593ff4327ee44fc74fad7043153fdc11c119f0d7048aadb9e173087b320df8"} Jul 24 15:43:50.833981 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.833919 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"e9ac2b8970c520d85be5b7e45ef414e4f33c346ba357d4b0986184198fdcd4e4"} Jul 24 15:43:50.833981 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.833947 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"8009932c2cf99d99534adc14fdd4159928a53d5a9e240b54b54fd05711e7522f"} Jul 24 15:43:50.833981 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.833972 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"71e228dbafee5d661fd36a12e85204b395c00495055bb36cdde786809c485f97"} Jul 24 15:43:50.834087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.833994 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"a6673d59d709e7b26f48f5b3a12f485a12cce48c1d2220292c9478177b59d1cc"} Jul 24 15:43:50.834087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.834019 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"7851499f-f916-47c4-8b6c-fe1e9a120c0b","Type":"ContainerStarted","Data":"94223acfa2cd0b0bab6d9ca63cde043857285c065f97a4ab4a2dcd773feece82"} Jul 24 15:43:50.851904 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.851818 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-c7f9fccb9-n4qzn" podStartSLOduration=2.197573283 podStartE2EDuration="3.851799601s" totalImagesPullingTime="1.654226318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:50.850219552 +0000 UTC m=+162.381807163" watchObservedRunningTime="2026-07-24 15:43:50.851799601 +0000 UTC m=+162.383387215" Jul 24 15:43:50.879457 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:50.879406 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.8551177709999997 podStartE2EDuration="2.87938981s" totalImagesPullingTime="0s" totalInitContainerRuntime="24.272039ms" isStatefulPod=false podCreationTimestamp="2026-07-24 15:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:43:50.876879044 +0000 UTC m=+162.408466672" watchObservedRunningTime="2026-07-24 15:43:50.87938981 +0000 UTC m=+162.410977422" Jul 24 15:43:54.184516 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:43:54.184485 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:44:49.184794 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:44:49.184763 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:44:49.200820 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:44:49.200796 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:44:50.091695 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:44:50.091668 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 24 15:46:08.859057 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:08.859030 2543 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 15:46:29.851717 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.851684 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx"] Jul 24 15:46:29.854819 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.854803 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:29.857067 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.857046 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 24 15:46:29.857837 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.857822 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-fg2vp\"" Jul 24 15:46:29.857929 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.857826 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 24 15:46:29.864410 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.864388 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx"] Jul 24 15:46:29.970116 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.970092 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168938dc-5f20-4cc9-9ac2-5708f206aa9d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vpvvx\" (UID: \"168938dc-5f20-4cc9-9ac2-5708f206aa9d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:29.970268 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:29.970181 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sg8m\" (UniqueName: \"kubernetes.io/projected/168938dc-5f20-4cc9-9ac2-5708f206aa9d-kube-api-access-5sg8m\") pod \"cert-manager-cainjector-559cdd8c87-vpvvx\" (UID: \"168938dc-5f20-4cc9-9ac2-5708f206aa9d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:30.070556 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.070524 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168938dc-5f20-4cc9-9ac2-5708f206aa9d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vpvvx\" (UID: \"168938dc-5f20-4cc9-9ac2-5708f206aa9d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:30.070724 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.070644 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sg8m\" (UniqueName: \"kubernetes.io/projected/168938dc-5f20-4cc9-9ac2-5708f206aa9d-kube-api-access-5sg8m\") pod \"cert-manager-cainjector-559cdd8c87-vpvvx\" (UID: \"168938dc-5f20-4cc9-9ac2-5708f206aa9d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:30.078274 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.078247 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/168938dc-5f20-4cc9-9ac2-5708f206aa9d-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vpvvx\" (UID: \"168938dc-5f20-4cc9-9ac2-5708f206aa9d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:30.078523 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.078503 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sg8m\" (UniqueName: \"kubernetes.io/projected/168938dc-5f20-4cc9-9ac2-5708f206aa9d-kube-api-access-5sg8m\") pod \"cert-manager-cainjector-559cdd8c87-vpvvx\" (UID: \"168938dc-5f20-4cc9-9ac2-5708f206aa9d\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:30.175315 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.175251 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" Jul 24 15:46:30.306835 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.306809 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx"] Jul 24 15:46:30.309292 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:46:30.309266 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod168938dc_5f20_4cc9_9ac2_5708f206aa9d.slice/crio-612d8a5355a88265d567a54940109ca8e15b0ced72974c1e96f52fde099e793d WatchSource:0}: Error finding container 612d8a5355a88265d567a54940109ca8e15b0ced72974c1e96f52fde099e793d: Status 404 returned error can't find the container with id 612d8a5355a88265d567a54940109ca8e15b0ced72974c1e96f52fde099e793d Jul 24 15:46:30.314469 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.314450 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:46:30.469102 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:30.469074 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" event={"ID":"168938dc-5f20-4cc9-9ac2-5708f206aa9d","Type":"ContainerStarted","Data":"612d8a5355a88265d567a54940109ca8e15b0ced72974c1e96f52fde099e793d"} Jul 24 15:46:33.484776 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:33.484741 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" event={"ID":"168938dc-5f20-4cc9-9ac2-5708f206aa9d","Type":"ContainerStarted","Data":"88ed4b3062ef3db12d7e39a00e2ff515146c531418abe94c1627b73c8498ee6d"} Jul 24 15:46:33.499297 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:33.499248 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vpvvx" podStartSLOduration=1.604155627 podStartE2EDuration="4.499217699s" totalImagesPullingTime="2.895062072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:33.498370283 +0000 UTC m=+325.029957897" watchObservedRunningTime="2026-07-24 15:46:33.499217699 +0000 UTC m=+325.030805311" Jul 24 15:46:36.579344 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.579308 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7h245"] Jul 24 15:46:36.582708 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.582689 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.584833 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.584816 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gzzvq\"" Jul 24 15:46:36.592204 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.592174 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7h245"] Jul 24 15:46:36.731221 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.731184 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj9l5\" (UniqueName: \"kubernetes.io/projected/17f371fb-ac3f-48a3-bac4-f8462e8359c5-kube-api-access-nj9l5\") pod \"cert-manager-55b4d954d5-7h245\" (UID: \"17f371fb-ac3f-48a3-bac4-f8462e8359c5\") " pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.731403 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.731227 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17f371fb-ac3f-48a3-bac4-f8462e8359c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-7h245\" (UID: \"17f371fb-ac3f-48a3-bac4-f8462e8359c5\") " pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.831999 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.831911 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj9l5\" (UniqueName: \"kubernetes.io/projected/17f371fb-ac3f-48a3-bac4-f8462e8359c5-kube-api-access-nj9l5\") pod \"cert-manager-55b4d954d5-7h245\" (UID: \"17f371fb-ac3f-48a3-bac4-f8462e8359c5\") " pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.831999 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.831962 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17f371fb-ac3f-48a3-bac4-f8462e8359c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-7h245\" (UID: \"17f371fb-ac3f-48a3-bac4-f8462e8359c5\") " pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.839943 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.839916 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/17f371fb-ac3f-48a3-bac4-f8462e8359c5-bound-sa-token\") pod \"cert-manager-55b4d954d5-7h245\" (UID: \"17f371fb-ac3f-48a3-bac4-f8462e8359c5\") " pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.840093 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.840069 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj9l5\" (UniqueName: \"kubernetes.io/projected/17f371fb-ac3f-48a3-bac4-f8462e8359c5-kube-api-access-nj9l5\") pod \"cert-manager-55b4d954d5-7h245\" (UID: \"17f371fb-ac3f-48a3-bac4-f8462e8359c5\") " pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:36.891799 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:36.891765 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-7h245" Jul 24 15:46:37.026098 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:37.025992 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-7h245"] Jul 24 15:46:37.028048 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:46:37.028017 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17f371fb_ac3f_48a3_bac4_f8462e8359c5.slice/crio-c49176db2df9154f8f8ea6f1d2cf7a0cbb6eadf38b91a3183abb869b735aa482 WatchSource:0}: Error finding container c49176db2df9154f8f8ea6f1d2cf7a0cbb6eadf38b91a3183abb869b735aa482: Status 404 returned error can't find the container with id c49176db2df9154f8f8ea6f1d2cf7a0cbb6eadf38b91a3183abb869b735aa482 Jul 24 15:46:37.503214 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:37.503180 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7h245" event={"ID":"17f371fb-ac3f-48a3-bac4-f8462e8359c5","Type":"ContainerStarted","Data":"ecfb43c4a68443acb0637a526121ffd6b3a17f407b8fa67caec20768acea2230"} Jul 24 15:46:37.503389 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:37.503226 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-7h245" event={"ID":"17f371fb-ac3f-48a3-bac4-f8462e8359c5","Type":"ContainerStarted","Data":"c49176db2df9154f8f8ea6f1d2cf7a0cbb6eadf38b91a3183abb869b735aa482"} Jul 24 15:46:37.519466 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:37.519416 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-7h245" podStartSLOduration=1.519401859 podStartE2EDuration="1.519401859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:46:37.517198402 +0000 UTC m=+329.048786015" watchObservedRunningTime="2026-07-24 15:46:37.519401859 +0000 UTC m=+329.050989471" Jul 24 15:46:59.142452 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.142420 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk"] Jul 24 15:46:59.144695 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.144665 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.147273 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.147250 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 15:46:59.147386 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.147330 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 15:46:59.147542 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.147525 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rkv9t\"" Jul 24 15:46:59.147636 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.147542 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 15:46:59.147690 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.147678 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 15:46:59.162643 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.162619 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk"] Jul 24 15:46:59.216028 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.216000 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwbfd\" (UniqueName: \"kubernetes.io/projected/7681e43a-352b-433f-882d-0daeb4f689ac-kube-api-access-bwbfd\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.216151 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.216036 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7681e43a-352b-433f-882d-0daeb4f689ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.216151 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.216054 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7681e43a-352b-433f-882d-0daeb4f689ac-manifests\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.216151 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.216071 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7681e43a-352b-433f-882d-0daeb4f689ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.216151 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.216133 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7681e43a-352b-433f-882d-0daeb4f689ac-tmp\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.317207 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.317177 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwbfd\" (UniqueName: \"kubernetes.io/projected/7681e43a-352b-433f-882d-0daeb4f689ac-kube-api-access-bwbfd\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.317367 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.317221 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7681e43a-352b-433f-882d-0daeb4f689ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.317367 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.317240 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7681e43a-352b-433f-882d-0daeb4f689ac-manifests\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.317367 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.317266 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7681e43a-352b-433f-882d-0daeb4f689ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.317367 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.317326 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7681e43a-352b-433f-882d-0daeb4f689ac-tmp\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.317640 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.317618 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7681e43a-352b-433f-882d-0daeb4f689ac-manifests\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.319817 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.319793 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7681e43a-352b-433f-882d-0daeb4f689ac-tmp\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.319910 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.319886 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7681e43a-352b-433f-882d-0daeb4f689ac-webhook-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.319994 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.319978 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7681e43a-352b-433f-882d-0daeb4f689ac-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.327527 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.327506 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwbfd\" (UniqueName: \"kubernetes.io/projected/7681e43a-352b-433f-882d-0daeb4f689ac-kube-api-access-bwbfd\") pod \"opendatahub-operator-controller-manager-6798866cc5-mdmsk\" (UID: \"7681e43a-352b-433f-882d-0daeb4f689ac\") " pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.456810 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.456729 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:46:59.594809 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:46:59.594766 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk"] Jul 24 15:46:59.597642 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:46:59.597611 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7681e43a_352b_433f_882d_0daeb4f689ac.slice/crio-104460d668c91d41eadf5dba5fb5eead83640e6e10a9ea61a960917ffd5ccae9 WatchSource:0}: Error finding container 104460d668c91d41eadf5dba5fb5eead83640e6e10a9ea61a960917ffd5ccae9: Status 404 returned error can't find the container with id 104460d668c91d41eadf5dba5fb5eead83640e6e10a9ea61a960917ffd5ccae9 Jul 24 15:47:00.607947 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:00.607906 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" event={"ID":"7681e43a-352b-433f-882d-0daeb4f689ac","Type":"ContainerStarted","Data":"104460d668c91d41eadf5dba5fb5eead83640e6e10a9ea61a960917ffd5ccae9"} Jul 24 15:47:02.619156 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:02.619117 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" event={"ID":"7681e43a-352b-433f-882d-0daeb4f689ac","Type":"ContainerStarted","Data":"282b3a725c9d76d183334e9e2b5e8e23cb2f6ac5951ee3e4c9f3bf54b2995193"} Jul 24 15:47:03.627014 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:03.626979 2543 generic.go:356] "Generic (PLEG): container finished" podID="7681e43a-352b-433f-882d-0daeb4f689ac" containerID="282b3a725c9d76d183334e9e2b5e8e23cb2f6ac5951ee3e4c9f3bf54b2995193" exitCode=0 Jul 24 15:47:03.627364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:03.627057 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" event={"ID":"7681e43a-352b-433f-882d-0daeb4f689ac","Type":"ContainerDied","Data":"282b3a725c9d76d183334e9e2b5e8e23cb2f6ac5951ee3e4c9f3bf54b2995193"} Jul 24 15:47:04.636014 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:04.635978 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" event={"ID":"7681e43a-352b-433f-882d-0daeb4f689ac","Type":"ContainerStarted","Data":"a72a43a33451c737d5812a0434b9aea729b4ee31bc9b44b4b460378d05ccff4c"} Jul 24 15:47:04.636636 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:04.636102 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:47:04.658496 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:04.658439 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" podStartSLOduration=1.377185229 podStartE2EDuration="5.658423226s" totalImagesPullingTime="3.25999915s" totalInitContainerRuntime="1.021238847s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:46:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:04.655903223 +0000 UTC m=+356.187490837" watchObservedRunningTime="2026-07-24 15:47:04.658423226 +0000 UTC m=+356.190010839" Jul 24 15:47:15.643421 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:15.643385 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6798866cc5-mdmsk" Jul 24 15:47:26.348946 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.348912 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb"] Jul 24 15:47:26.362185 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.362151 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.365705 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.365675 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 15:47:26.366072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.366046 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 15:47:26.366206 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.366133 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-4q599\"" Jul 24 15:47:26.366400 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.366387 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 15:47:26.381366 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.381343 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb"] Jul 24 15:47:26.463446 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463417 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463455 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463473 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463509 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463553 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463630 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463654 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463648 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463888 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463700 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pht\" (UniqueName: \"kubernetes.io/projected/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-kube-api-access-s5pht\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.463888 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.463735 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564147 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564110 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564147 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564146 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564175 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564196 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564240 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564258 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564391 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564285 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pht\" (UniqueName: \"kubernetes.io/projected/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-kube-api-access-s5pht\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564656 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564522 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564707 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564655 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564707 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564697 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564801 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564721 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564897 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564877 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.564950 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.564922 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.565144 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.565128 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.566666 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.566641 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.567102 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.567086 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.575078 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.575054 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.575485 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.575469 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pht\" (UniqueName: \"kubernetes.io/projected/7e83ef7d-6413-4a7a-ad0a-d19b163e9f31-kube-api-access-s5pht\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb\" (UID: \"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.673317 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.673056 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:26.889364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:26.889334 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb"] Jul 24 15:47:26.889905 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:47:26.889875 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e83ef7d_6413_4a7a_ad0a_d19b163e9f31.slice/crio-b88e4d727b327966333fa1b67345f06f6c8f0dba1ab650897c9f5a558e91a62e WatchSource:0}: Error finding container b88e4d727b327966333fa1b67345f06f6c8f0dba1ab650897c9f5a558e91a62e: Status 404 returned error can't find the container with id b88e4d727b327966333fa1b67345f06f6c8f0dba1ab650897c9f5a558e91a62e Jul 24 15:47:27.748381 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:27.748196 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" event={"ID":"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31","Type":"ContainerStarted","Data":"b88e4d727b327966333fa1b67345f06f6c8f0dba1ab650897c9f5a558e91a62e"} Jul 24 15:47:28.516493 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.516450 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch"] Jul 24 15:47:28.519030 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.519006 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.521869 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.521845 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 15:47:28.522196 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.522166 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 15:47:28.523216 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.523185 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 15:47:28.523310 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.523275 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-j5m9b\"" Jul 24 15:47:28.523362 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.523325 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 15:47:28.523779 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.523759 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 15:47:28.543767 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.543745 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch"] Jul 24 15:47:28.593015 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.592977 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32ebc5e8-20f8-449d-8868-c52e760e578f-manager-config\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.593193 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.593057 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9fsg\" (UniqueName: \"kubernetes.io/projected/32ebc5e8-20f8-449d-8868-c52e760e578f-kube-api-access-r9fsg\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.593193 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.593123 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32ebc5e8-20f8-449d-8868-c52e760e578f-cert\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.593193 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.593153 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32ebc5e8-20f8-449d-8868-c52e760e578f-metrics-cert\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.694454 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.694423 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9fsg\" (UniqueName: \"kubernetes.io/projected/32ebc5e8-20f8-449d-8868-c52e760e578f-kube-api-access-r9fsg\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.694658 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.694491 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32ebc5e8-20f8-449d-8868-c52e760e578f-cert\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.694658 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.694516 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32ebc5e8-20f8-449d-8868-c52e760e578f-metrics-cert\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.694658 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.694616 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32ebc5e8-20f8-449d-8868-c52e760e578f-manager-config\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.697285 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.697263 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32ebc5e8-20f8-449d-8868-c52e760e578f-cert\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.697516 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.697471 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/32ebc5e8-20f8-449d-8868-c52e760e578f-manager-config\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.697737 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.697716 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/32ebc5e8-20f8-449d-8868-c52e760e578f-metrics-cert\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.729779 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.729752 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9fsg\" (UniqueName: \"kubernetes.io/projected/32ebc5e8-20f8-449d-8868-c52e760e578f-kube-api-access-r9fsg\") pod \"lws-controller-manager-65dbfbfd5-v9wch\" (UID: \"32ebc5e8-20f8-449d-8868-c52e760e578f\") " pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:28.832369 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:28.832294 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:29.220541 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.220504 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j8znp"] Jul 24 15:47:29.223208 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.223187 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.226364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.226342 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 15:47:29.226364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.226351 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qctrs\"" Jul 24 15:47:29.228463 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.228440 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz"] Jul 24 15:47:29.230959 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.230938 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.234475 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.234051 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-7ns7t\"" Jul 24 15:47:29.234475 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.234245 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 15:47:29.235757 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.235734 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j8znp"] Jul 24 15:47:29.251735 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.251710 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz"] Jul 24 15:47:29.300577 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.300533 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/691e4d01-122f-48e0-a3d8-d055cdd619f2-cert\") pod \"kserve-controller-manager-d6f4b65b8-j8znp\" (UID: \"691e4d01-122f-48e0-a3d8-d055cdd619f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.300771 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.300652 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvfnl\" (UniqueName: \"kubernetes.io/projected/691e4d01-122f-48e0-a3d8-d055cdd619f2-kube-api-access-xvfnl\") pod \"kserve-controller-manager-d6f4b65b8-j8znp\" (UID: \"691e4d01-122f-48e0-a3d8-d055cdd619f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.401703 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.401675 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvfnl\" (UniqueName: \"kubernetes.io/projected/691e4d01-122f-48e0-a3d8-d055cdd619f2-kube-api-access-xvfnl\") pod \"kserve-controller-manager-d6f4b65b8-j8znp\" (UID: \"691e4d01-122f-48e0-a3d8-d055cdd619f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.401892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.401761 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/691e4d01-122f-48e0-a3d8-d055cdd619f2-cert\") pod \"kserve-controller-manager-d6f4b65b8-j8znp\" (UID: \"691e4d01-122f-48e0-a3d8-d055cdd619f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.401892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.401792 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg425\" (UniqueName: \"kubernetes.io/projected/b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb-kube-api-access-dg425\") pod \"llmisvc-controller-manager-777c6bd7d5-frnpz\" (UID: \"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.401892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.401825 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-frnpz\" (UID: \"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.404601 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.404556 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/691e4d01-122f-48e0-a3d8-d055cdd619f2-cert\") pod \"kserve-controller-manager-d6f4b65b8-j8znp\" (UID: \"691e4d01-122f-48e0-a3d8-d055cdd619f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.410594 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.410546 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvfnl\" (UniqueName: \"kubernetes.io/projected/691e4d01-122f-48e0-a3d8-d055cdd619f2-kube-api-access-xvfnl\") pod \"kserve-controller-manager-d6f4b65b8-j8znp\" (UID: \"691e4d01-122f-48e0-a3d8-d055cdd619f2\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.503087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.502995 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg425\" (UniqueName: \"kubernetes.io/projected/b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb-kube-api-access-dg425\") pod \"llmisvc-controller-manager-777c6bd7d5-frnpz\" (UID: \"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.503087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.503083 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-frnpz\" (UID: \"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.506477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.506453 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-frnpz\" (UID: \"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.528309 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.528284 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg425\" (UniqueName: \"kubernetes.io/projected/b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb-kube-api-access-dg425\") pod \"llmisvc-controller-manager-777c6bd7d5-frnpz\" (UID: \"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.541142 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.541072 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:29.548647 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.548623 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:29.747665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.747485 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:47:29.747665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.747596 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:47:29.747665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.747635 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:47:29.920316 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.920179 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch"] Jul 24 15:47:29.925070 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:47:29.924951 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ebc5e8_20f8_449d_8868_c52e760e578f.slice/crio-2a931b6eb6f2e6f88e3ce371edfccb330c9434ffb19ae36f9de78df532ba8bb6 WatchSource:0}: Error finding container 2a931b6eb6f2e6f88e3ce371edfccb330c9434ffb19ae36f9de78df532ba8bb6: Status 404 returned error can't find the container with id 2a931b6eb6f2e6f88e3ce371edfccb330c9434ffb19ae36f9de78df532ba8bb6 Jul 24 15:47:29.944533 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.944509 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-j8znp"] Jul 24 15:47:29.947202 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:47:29.947166 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod691e4d01_122f_48e0_a3d8_d055cdd619f2.slice/crio-37ec8ab69669c5f4b57dd464cd1c52d8bcc416639616aba0e4d786828e2d6ed7 WatchSource:0}: Error finding container 37ec8ab69669c5f4b57dd464cd1c52d8bcc416639616aba0e4d786828e2d6ed7: Status 404 returned error can't find the container with id 37ec8ab69669c5f4b57dd464cd1c52d8bcc416639616aba0e4d786828e2d6ed7 Jul 24 15:47:29.964269 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:29.964246 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz"] Jul 24 15:47:29.965943 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:47:29.965916 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb6be9a8e_6d4a_4733_97c6_e4fc4e7386eb.slice/crio-7081f4fc4dda312d42ff42ac9523bd1d3d8611b7db8749dd95b7dddd460670d5 WatchSource:0}: Error finding container 7081f4fc4dda312d42ff42ac9523bd1d3d8611b7db8749dd95b7dddd460670d5: Status 404 returned error can't find the container with id 7081f4fc4dda312d42ff42ac9523bd1d3d8611b7db8749dd95b7dddd460670d5 Jul 24 15:47:30.779091 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:30.778991 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" event={"ID":"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb","Type":"ContainerStarted","Data":"7081f4fc4dda312d42ff42ac9523bd1d3d8611b7db8749dd95b7dddd460670d5"} Jul 24 15:47:30.782411 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:30.782377 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" event={"ID":"691e4d01-122f-48e0-a3d8-d055cdd619f2","Type":"ContainerStarted","Data":"37ec8ab69669c5f4b57dd464cd1c52d8bcc416639616aba0e4d786828e2d6ed7"} Jul 24 15:47:30.787703 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:30.787637 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" event={"ID":"32ebc5e8-20f8-449d-8868-c52e760e578f","Type":"ContainerStarted","Data":"2a931b6eb6f2e6f88e3ce371edfccb330c9434ffb19ae36f9de78df532ba8bb6"} Jul 24 15:47:30.796186 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:30.795835 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" event={"ID":"7e83ef7d-6413-4a7a-ad0a-d19b163e9f31","Type":"ContainerStarted","Data":"493ad1f95d1362035654e5d4bab958aa0dbec9cfb08944c515d261909d91942e"} Jul 24 15:47:30.840640 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:30.840528 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" podStartSLOduration=1.985570095 podStartE2EDuration="4.84050428s" totalImagesPullingTime="2.854934185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:30.839201003 +0000 UTC m=+382.370788664" watchObservedRunningTime="2026-07-24 15:47:30.84050428 +0000 UTC m=+382.372091898" Jul 24 15:47:31.674416 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:31.674377 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:31.680120 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:31.680098 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:31.802150 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:31.802090 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:31.803296 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:31.803274 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb" Jul 24 15:47:33.816346 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:33.816310 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" event={"ID":"691e4d01-122f-48e0-a3d8-d055cdd619f2","Type":"ContainerStarted","Data":"76882bb93b47b0dbc10a035ae91b0fb036891ce19c83dff20682068cbb58cf72"} Jul 24 15:47:33.816841 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:33.816428 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:47:33.818329 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:33.818270 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" event={"ID":"32ebc5e8-20f8-449d-8868-c52e760e578f","Type":"ContainerStarted","Data":"106e9655f287c554335739d1536828d27d48a5e979290164de51aa8ff54621e1"} Jul 24 15:47:33.818506 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:33.818491 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:47:33.833124 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:33.833069 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" podStartSLOduration=1.531028956 podStartE2EDuration="4.833048725s" totalImagesPullingTime="3.302019769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:33.831653908 +0000 UTC m=+385.363241533" watchObservedRunningTime="2026-07-24 15:47:33.833048725 +0000 UTC m=+385.364636342" Jul 24 15:47:33.849825 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:33.849778 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" podStartSLOduration=2.453870367 podStartE2EDuration="5.849763692s" totalImagesPullingTime="3.395893325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:33.848008635 +0000 UTC m=+385.379596275" watchObservedRunningTime="2026-07-24 15:47:33.849763692 +0000 UTC m=+385.381351305" Jul 24 15:47:34.824892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:34.824855 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" event={"ID":"b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb","Type":"ContainerStarted","Data":"a8a95fc3aecfbd0ba08061f94998ea37d079a8ca959fff73c39b4b2a26f555d6"} Jul 24 15:47:34.841287 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:34.841244 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" podStartSLOduration=1.7164287630000001 podStartE2EDuration="5.841230078s" totalImagesPullingTime="4.124801315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:47:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:47:34.838793021 +0000 UTC m=+386.370380635" watchObservedRunningTime="2026-07-24 15:47:34.841230078 +0000 UTC m=+386.372817751" Jul 24 15:47:35.829832 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:35.829794 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:47:44.827293 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:47:44.827267 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-65dbfbfd5-v9wch" Jul 24 15:48:04.829964 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:04.829934 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-j8znp" Jul 24 15:48:06.836521 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:06.836486 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-frnpz" Jul 24 15:48:19.588259 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.588224 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lgrdf"] Jul 24 15:48:19.597282 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.597258 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:19.600138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.600077 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:48:19.600861 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.600839 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:48:19.601152 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.601017 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-pn9ll\"" Jul 24 15:48:19.601882 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.601857 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lgrdf"] Jul 24 15:48:19.661043 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.661014 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbz6\" (UniqueName: \"kubernetes.io/projected/bce970e4-3602-4af1-8852-5f129aa94e8f-kube-api-access-cbbz6\") pod \"kuadrant-operator-catalog-lgrdf\" (UID: \"bce970e4-3602-4af1-8852-5f129aa94e8f\") " pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:19.761486 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.761454 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbz6\" (UniqueName: \"kubernetes.io/projected/bce970e4-3602-4af1-8852-5f129aa94e8f-kube-api-access-cbbz6\") pod \"kuadrant-operator-catalog-lgrdf\" (UID: \"bce970e4-3602-4af1-8852-5f129aa94e8f\") " pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:19.769683 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.769658 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbz6\" (UniqueName: \"kubernetes.io/projected/bce970e4-3602-4af1-8852-5f129aa94e8f-kube-api-access-cbbz6\") pod \"kuadrant-operator-catalog-lgrdf\" (UID: \"bce970e4-3602-4af1-8852-5f129aa94e8f\") " pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:19.908644 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.908536 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:19.954421 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:19.954389 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lgrdf"] Jul 24 15:48:20.037084 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:20.037060 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lgrdf"] Jul 24 15:48:20.039671 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:48:20.039639 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce970e4_3602_4af1_8852_5f129aa94e8f.slice/crio-734cc93084cc816a24e25dd40fa419eb34b4fdfa44c80a7ab8b0e4ce5210b523 WatchSource:0}: Error finding container 734cc93084cc816a24e25dd40fa419eb34b4fdfa44c80a7ab8b0e4ce5210b523: Status 404 returned error can't find the container with id 734cc93084cc816a24e25dd40fa419eb34b4fdfa44c80a7ab8b0e4ce5210b523 Jul 24 15:48:21.046625 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:21.046589 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" event={"ID":"bce970e4-3602-4af1-8852-5f129aa94e8f","Type":"ContainerStarted","Data":"734cc93084cc816a24e25dd40fa419eb34b4fdfa44c80a7ab8b0e4ce5210b523"} Jul 24 15:48:22.047525 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.047492 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" event={"ID":"bce970e4-3602-4af1-8852-5f129aa94e8f","Type":"ContainerStarted","Data":"3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9"} Jul 24 15:48:22.048019 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.047619 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" podUID="bce970e4-3602-4af1-8852-5f129aa94e8f" containerName="registry-server" containerID="cri-o://3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9" gracePeriod=2 Jul 24 15:48:22.080234 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.080185 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" podStartSLOduration=1.433918139 podStartE2EDuration="3.08016839s" totalImagesPullingTime="1.646250251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:22.079523926 +0000 UTC m=+433.611111559" watchObservedRunningTime="2026-07-24 15:48:22.08016839 +0000 UTC m=+433.611755980" Jul 24 15:48:22.293692 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.293671 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:22.388164 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.388075 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cbbz6\" (UniqueName: \"kubernetes.io/projected/bce970e4-3602-4af1-8852-5f129aa94e8f-kube-api-access-cbbz6\") pod \"bce970e4-3602-4af1-8852-5f129aa94e8f\" (UID: \"bce970e4-3602-4af1-8852-5f129aa94e8f\") " Jul 24 15:48:22.390404 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.390375 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce970e4-3602-4af1-8852-5f129aa94e8f-kube-api-access-cbbz6" (OuterVolumeSpecName: "kube-api-access-cbbz6") pod "bce970e4-3602-4af1-8852-5f129aa94e8f" (UID: "bce970e4-3602-4af1-8852-5f129aa94e8f"). InnerVolumeSpecName "kube-api-access-cbbz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:48:22.489247 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:22.489214 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbbz6\" (UniqueName: \"kubernetes.io/projected/bce970e4-3602-4af1-8852-5f129aa94e8f-kube-api-access-cbbz6\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:48:23.053237 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.053201 2543 generic.go:356] "Generic (PLEG): container finished" podID="bce970e4-3602-4af1-8852-5f129aa94e8f" containerID="3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9" exitCode=0 Jul 24 15:48:23.053669 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.053242 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" event={"ID":"bce970e4-3602-4af1-8852-5f129aa94e8f","Type":"ContainerDied","Data":"3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9"} Jul 24 15:48:23.053669 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.053260 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" Jul 24 15:48:23.053669 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.053300 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-lgrdf" event={"ID":"bce970e4-3602-4af1-8852-5f129aa94e8f","Type":"ContainerDied","Data":"734cc93084cc816a24e25dd40fa419eb34b4fdfa44c80a7ab8b0e4ce5210b523"} Jul 24 15:48:23.053669 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.053324 2543 scope.go:117] "RemoveContainer" containerID="3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9" Jul 24 15:48:23.062276 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.062261 2543 scope.go:117] "RemoveContainer" containerID="3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9" Jul 24 15:48:23.062536 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:48:23.062513 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9\": container with ID starting with 3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9 not found: ID does not exist" containerID="3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9" Jul 24 15:48:23.062643 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.062542 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9"} err="failed to get container status \"3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9\": rpc error: code = NotFound desc = could not find container \"3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9\": container with ID starting with 3fca62178a8370c99443e41bb8afc82dbc2647b1da1bf3945dd9cfb37fac76f9 not found: ID does not exist" Jul 24 15:48:23.075274 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.075247 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lgrdf"] Jul 24 15:48:23.079358 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:23.079337 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-lgrdf"] Jul 24 15:48:25.043296 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:25.043266 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce970e4-3602-4af1-8852-5f129aa94e8f" path="/var/lib/kubelet/pods/bce970e4-3602-4af1-8852-5f129aa94e8f/volumes" Jul 24 15:48:50.435097 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.435066 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8"] Jul 24 15:48:50.435664 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.435645 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce970e4-3602-4af1-8852-5f129aa94e8f" containerName="registry-server" Jul 24 15:48:50.435741 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.435666 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce970e4-3602-4af1-8852-5f129aa94e8f" containerName="registry-server" Jul 24 15:48:50.435796 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.435766 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce970e4-3602-4af1-8852-5f129aa94e8f" containerName="registry-server" Jul 24 15:48:50.437828 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.437805 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:48:50.441794 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.441776 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 15:48:50.442975 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.442959 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 24 15:48:50.443086 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.442959 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-6z749\"" Jul 24 15:48:50.443086 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.442959 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 15:48:50.452877 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.452859 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8"] Jul 24 15:48:50.540536 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.540504 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x8fr\" (UniqueName: \"kubernetes.io/projected/38aba10a-872b-4c49-a0d0-ceffeb0ed4eb-kube-api-access-8x8fr\") pod \"dns-operator-controller-manager-864dd78c4c-9ppx8\" (UID: \"38aba10a-872b-4c49-a0d0-ceffeb0ed4eb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:48:50.641539 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.641500 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x8fr\" (UniqueName: \"kubernetes.io/projected/38aba10a-872b-4c49-a0d0-ceffeb0ed4eb-kube-api-access-8x8fr\") pod \"dns-operator-controller-manager-864dd78c4c-9ppx8\" (UID: \"38aba10a-872b-4c49-a0d0-ceffeb0ed4eb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:48:50.651549 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.651512 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x8fr\" (UniqueName: \"kubernetes.io/projected/38aba10a-872b-4c49-a0d0-ceffeb0ed4eb-kube-api-access-8x8fr\") pod \"dns-operator-controller-manager-864dd78c4c-9ppx8\" (UID: \"38aba10a-872b-4c49-a0d0-ceffeb0ed4eb\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:48:50.747978 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.747918 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:48:50.897137 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:50.897110 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8"] Jul 24 15:48:50.899215 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:48:50.899190 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38aba10a_872b_4c49_a0d0_ceffeb0ed4eb.slice/crio-6ec22fdf6449d35449aea2e52a880c0a9cc1f425b06cfdcf792fe2fff17abe8d WatchSource:0}: Error finding container 6ec22fdf6449d35449aea2e52a880c0a9cc1f425b06cfdcf792fe2fff17abe8d: Status 404 returned error can't find the container with id 6ec22fdf6449d35449aea2e52a880c0a9cc1f425b06cfdcf792fe2fff17abe8d Jul 24 15:48:51.179484 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:51.179397 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" event={"ID":"38aba10a-872b-4c49-a0d0-ceffeb0ed4eb","Type":"ContainerStarted","Data":"6ec22fdf6449d35449aea2e52a880c0a9cc1f425b06cfdcf792fe2fff17abe8d"} Jul 24 15:48:52.126269 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.126237 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8b555fb48-gf8p4"] Jul 24 15:48:52.130346 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.130324 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.135477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.135235 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 24 15:48:52.135477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.135252 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vx7mq\"" Jul 24 15:48:52.135477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.135235 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 24 15:48:52.135477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.135317 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 24 15:48:52.135477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.135277 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 24 15:48:52.135477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.135235 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 24 15:48:52.143482 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.143462 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 24 15:48:52.153965 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.153942 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b555fb48-gf8p4"] Jul 24 15:48:52.264371 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264339 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-trusted-ca-bundle\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.264526 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264379 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-oauth-serving-cert\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.264526 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264410 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-serving-cert\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.264526 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264481 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-service-ca\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.264526 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264499 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxmcm\" (UniqueName: \"kubernetes.io/projected/ff644d32-acac-4aa5-ac75-244de09a7d2e-kube-api-access-qxmcm\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.264731 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264527 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-oauth-config\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.264731 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.264551 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-config\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.365896 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.365863 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-trusted-ca-bundle\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.365896 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.365897 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-oauth-serving-cert\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366131 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.365937 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-serving-cert\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366131 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.365992 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-service-ca\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366131 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366024 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxmcm\" (UniqueName: \"kubernetes.io/projected/ff644d32-acac-4aa5-ac75-244de09a7d2e-kube-api-access-qxmcm\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366131 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366069 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-oauth-config\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366131 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366088 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-config\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366732 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366704 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-oauth-serving-cert\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366855 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366753 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-service-ca\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366925 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366906 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-trusted-ca-bundle\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.366968 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.366906 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-config\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.368682 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.368657 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-oauth-config\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.368682 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.368667 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ff644d32-acac-4aa5-ac75-244de09a7d2e-console-serving-cert\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.380194 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.380125 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxmcm\" (UniqueName: \"kubernetes.io/projected/ff644d32-acac-4aa5-ac75-244de09a7d2e-kube-api-access-qxmcm\") pod \"console-8b555fb48-gf8p4\" (UID: \"ff644d32-acac-4aa5-ac75-244de09a7d2e\") " pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.443556 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.443524 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:48:52.608892 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:52.608868 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8b555fb48-gf8p4"] Jul 24 15:48:52.807952 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:48:52.807911 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff644d32_acac_4aa5_ac75_244de09a7d2e.slice/crio-15a62a46710a84337d7fb729a018c89832bb5a9f83e6d7d6a9d093172268ad7d WatchSource:0}: Error finding container 15a62a46710a84337d7fb729a018c89832bb5a9f83e6d7d6a9d093172268ad7d: Status 404 returned error can't find the container with id 15a62a46710a84337d7fb729a018c89832bb5a9f83e6d7d6a9d093172268ad7d Jul 24 15:48:53.194126 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:53.194023 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" event={"ID":"38aba10a-872b-4c49-a0d0-ceffeb0ed4eb","Type":"ContainerStarted","Data":"895d36f244363949cdc161f0c0a2d554ef50c00d4d75c31b1dacc99891a7fae5"} Jul 24 15:48:53.194638 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:53.194134 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:48:53.195703 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:53.195682 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b555fb48-gf8p4" event={"ID":"ff644d32-acac-4aa5-ac75-244de09a7d2e","Type":"ContainerStarted","Data":"226a41bc6cd60fe7fa5bfdfabd6f253ddd763139ac9a9b98a387a974c126a9f7"} Jul 24 15:48:53.195796 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:53.195718 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8b555fb48-gf8p4" event={"ID":"ff644d32-acac-4aa5-ac75-244de09a7d2e","Type":"ContainerStarted","Data":"15a62a46710a84337d7fb729a018c89832bb5a9f83e6d7d6a9d093172268ad7d"} Jul 24 15:48:53.220505 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:53.220448 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" podStartSLOduration=1.261768511 podStartE2EDuration="3.220430439s" totalImagesPullingTime="1.958661928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:53.219979482 +0000 UTC m=+464.751567096" watchObservedRunningTime="2026-07-24 15:48:53.220430439 +0000 UTC m=+464.752018053" Jul 24 15:48:53.257375 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:53.257325 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8b555fb48-gf8p4" podStartSLOduration=1.257306021 podStartE2EDuration="1.257306021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:48:53.255610835 +0000 UTC m=+464.787198475" watchObservedRunningTime="2026-07-24 15:48:53.257306021 +0000 UTC m=+464.788893634" Jul 24 15:48:55.197995 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.197969 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d"] Jul 24 15:48:55.201550 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.201532 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.204696 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.204667 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wq5kn\"" Jul 24 15:48:55.233267 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.233244 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d"] Jul 24 15:48:55.393356 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.393324 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7wv7\" (UniqueName: \"kubernetes.io/projected/b27e5080-0269-423f-a949-417866553313-kube-api-access-g7wv7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.393547 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.393384 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27e5080-0269-423f-a949-417866553313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.494616 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.494587 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7wv7\" (UniqueName: \"kubernetes.io/projected/b27e5080-0269-423f-a949-417866553313-kube-api-access-g7wv7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.494769 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.494644 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27e5080-0269-423f-a949-417866553313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.495022 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.495003 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27e5080-0269-423f-a949-417866553313-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.522274 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.522251 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7wv7\" (UniqueName: \"kubernetes.io/projected/b27e5080-0269-423f-a949-417866553313-kube-api-access-g7wv7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.812235 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.812140 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:48:55.995168 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:55.995097 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d"] Jul 24 15:48:55.997488 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:48:55.997461 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb27e5080_0269_423f_a949_417866553313.slice/crio-c25c98e764a7fbf8fe7c19f2bf05962a794f3134c80447c750bc2d3adbe9f941 WatchSource:0}: Error finding container c25c98e764a7fbf8fe7c19f2bf05962a794f3134c80447c750bc2d3adbe9f941: Status 404 returned error can't find the container with id c25c98e764a7fbf8fe7c19f2bf05962a794f3134c80447c750bc2d3adbe9f941 Jul 24 15:48:56.213023 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:48:56.212948 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" event={"ID":"b27e5080-0269-423f-a949-417866553313","Type":"ContainerStarted","Data":"c25c98e764a7fbf8fe7c19f2bf05962a794f3134c80447c750bc2d3adbe9f941"} Jul 24 15:49:01.241556 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:01.241510 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" event={"ID":"b27e5080-0269-423f-a949-417866553313","Type":"ContainerStarted","Data":"8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6"} Jul 24 15:49:01.241952 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:01.241600 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:49:01.272626 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:01.272582 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" podStartSLOduration=1.699532927 podStartE2EDuration="6.272551443s" totalImagesPullingTime="4.573018516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:48:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:01.269727238 +0000 UTC m=+472.801314851" watchObservedRunningTime="2026-07-24 15:49:01.272551443 +0000 UTC m=+472.804139057" Jul 24 15:49:02.443724 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:02.443693 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:49:02.444206 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:02.443753 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:49:02.448438 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:02.448420 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:49:03.255425 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:03.255391 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8b555fb48-gf8p4" Jul 24 15:49:04.203316 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:04.203290 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9ppx8" Jul 24 15:49:12.249225 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:12.249188 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:49:14.369521 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.369488 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d"] Jul 24 15:49:14.370033 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.369677 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" podUID="b27e5080-0269-423f-a949-417866553313" containerName="manager" containerID="cri-o://8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6" gracePeriod=2 Jul 24 15:49:14.375293 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.375264 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d"] Jul 24 15:49:14.429724 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.429700 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j"] Jul 24 15:49:14.430102 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.430089 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b27e5080-0269-423f-a949-417866553313" containerName="manager" Jul 24 15:49:14.430148 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.430105 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b27e5080-0269-423f-a949-417866553313" containerName="manager" Jul 24 15:49:14.430201 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.430192 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b27e5080-0269-423f-a949-417866553313" containerName="manager" Jul 24 15:49:14.433323 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.433309 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.496323 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.496289 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j"] Jul 24 15:49:14.511867 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:49:14.511831 2543 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="b27e5080-0269-423f-a949-417866553313" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:49:14.572128 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.572094 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a660dd-5006-4f65-92bf-fddaf01d388f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zdc7j\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.572291 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.572186 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8knk\" (UniqueName: \"kubernetes.io/projected/b4a660dd-5006-4f65-92bf-fddaf01d388f-kube-api-access-q8knk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zdc7j\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.621889 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.621837 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:49:14.624363 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:49:14.624339 2543 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-tb25d\" is forbidden: User \"system:node:ip-10-0-143-210.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-143-210.ec2.internal' and this object" podUID="b27e5080-0269-423f-a949-417866553313" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:49:14.673166 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.673140 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7wv7\" (UniqueName: \"kubernetes.io/projected/b27e5080-0269-423f-a949-417866553313-kube-api-access-g7wv7\") pod \"b27e5080-0269-423f-a949-417866553313\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " Jul 24 15:49:14.673304 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.673180 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27e5080-0269-423f-a949-417866553313-extensions-socket-volume\") pod \"b27e5080-0269-423f-a949-417866553313\" (UID: \"b27e5080-0269-423f-a949-417866553313\") " Jul 24 15:49:14.673379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.673364 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a660dd-5006-4f65-92bf-fddaf01d388f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zdc7j\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.673442 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.673414 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8knk\" (UniqueName: \"kubernetes.io/projected/b4a660dd-5006-4f65-92bf-fddaf01d388f-kube-api-access-q8knk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zdc7j\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.673599 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.673548 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b27e5080-0269-423f-a949-417866553313-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b27e5080-0269-423f-a949-417866553313" (UID: "b27e5080-0269-423f-a949-417866553313"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:49:14.673751 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.673730 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a660dd-5006-4f65-92bf-fddaf01d388f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zdc7j\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.675378 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.675352 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b27e5080-0269-423f-a949-417866553313-kube-api-access-g7wv7" (OuterVolumeSpecName: "kube-api-access-g7wv7") pod "b27e5080-0269-423f-a949-417866553313" (UID: "b27e5080-0269-423f-a949-417866553313"). InnerVolumeSpecName "kube-api-access-g7wv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:49:14.683087 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.683065 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8knk\" (UniqueName: \"kubernetes.io/projected/b4a660dd-5006-4f65-92bf-fddaf01d388f-kube-api-access-q8knk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-zdc7j\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.757012 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.756979 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:14.774800 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.774774 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7wv7\" (UniqueName: \"kubernetes.io/projected/b27e5080-0269-423f-a949-417866553313-kube-api-access-g7wv7\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:49:14.774882 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.774807 2543 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b27e5080-0269-423f-a949-417866553313-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:49:14.910204 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:14.910180 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j"] Jul 24 15:49:14.912315 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:49:14.912290 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a660dd_5006_4f65_92bf_fddaf01d388f.slice/crio-70152b18aca99d4a5d6c942cc2e1af7eea1950f8a391f1863ed70020a2de75a1 WatchSource:0}: Error finding container 70152b18aca99d4a5d6c942cc2e1af7eea1950f8a391f1863ed70020a2de75a1: Status 404 returned error can't find the container with id 70152b18aca99d4a5d6c942cc2e1af7eea1950f8a391f1863ed70020a2de75a1 Jul 24 15:49:15.043463 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.043422 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b27e5080-0269-423f-a949-417866553313" path="/var/lib/kubelet/pods/b27e5080-0269-423f-a949-417866553313/volumes" Jul 24 15:49:15.312772 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.312732 2543 generic.go:356] "Generic (PLEG): container finished" podID="b27e5080-0269-423f-a949-417866553313" containerID="8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6" exitCode=0 Jul 24 15:49:15.312772 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.312775 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-tb25d" Jul 24 15:49:15.313005 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.312828 2543 scope.go:117] "RemoveContainer" containerID="8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6" Jul 24 15:49:15.314765 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.314740 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" event={"ID":"b4a660dd-5006-4f65-92bf-fddaf01d388f","Type":"ContainerStarted","Data":"6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae"} Jul 24 15:49:15.314890 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.314793 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" event={"ID":"b4a660dd-5006-4f65-92bf-fddaf01d388f","Type":"ContainerStarted","Data":"70152b18aca99d4a5d6c942cc2e1af7eea1950f8a391f1863ed70020a2de75a1"} Jul 24 15:49:15.314972 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.314958 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:15.321972 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.321957 2543 scope.go:117] "RemoveContainer" containerID="8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6" Jul 24 15:49:15.322199 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:49:15.322182 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6\": container with ID starting with 8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6 not found: ID does not exist" containerID="8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6" Jul 24 15:49:15.322245 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.322206 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6"} err="failed to get container status \"8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6\": rpc error: code = NotFound desc = could not find container \"8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6\": container with ID starting with 8dd8f75d88680c55b7ec0768a32f7b5f662ca149f32b061ce15608ae68e98bf6 not found: ID does not exist" Jul 24 15:49:15.346107 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.346064 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" podStartSLOduration=1.346052667 podStartE2EDuration="1.346052667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:15.344954734 +0000 UTC m=+486.876542366" watchObservedRunningTime="2026-07-24 15:49:15.346052667 +0000 UTC m=+486.877640280" Jul 24 15:49:15.618041 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.617969 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj"] Jul 24 15:49:15.622615 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.622597 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.654451 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.654424 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj"] Jul 24 15:49:15.684469 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.684440 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae2059d-1ccb-4caf-a4c8-a110172b594b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-w87cj\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.684615 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.684518 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpbnq\" (UniqueName: \"kubernetes.io/projected/3ae2059d-1ccb-4caf-a4c8-a110172b594b-kube-api-access-bpbnq\") pod \"kuadrant-operator-controller-manager-86878c769f-w87cj\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.784948 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.784913 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae2059d-1ccb-4caf-a4c8-a110172b594b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-w87cj\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.785101 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.784977 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpbnq\" (UniqueName: \"kubernetes.io/projected/3ae2059d-1ccb-4caf-a4c8-a110172b594b-kube-api-access-bpbnq\") pod \"kuadrant-operator-controller-manager-86878c769f-w87cj\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.785314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.785297 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae2059d-1ccb-4caf-a4c8-a110172b594b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-w87cj\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.800554 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.800529 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpbnq\" (UniqueName: \"kubernetes.io/projected/3ae2059d-1ccb-4caf-a4c8-a110172b594b-kube-api-access-bpbnq\") pod \"kuadrant-operator-controller-manager-86878c769f-w87cj\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:15.933350 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:15.933276 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:16.102663 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:16.102637 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj"] Jul 24 15:49:16.105321 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:49:16.105297 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae2059d_1ccb_4caf_a4c8_a110172b594b.slice/crio-7e5dcaf05a83eb79fec6ace1be66d51fd00fd4c966452a6b09521e7ea8aeb72b WatchSource:0}: Error finding container 7e5dcaf05a83eb79fec6ace1be66d51fd00fd4c966452a6b09521e7ea8aeb72b: Status 404 returned error can't find the container with id 7e5dcaf05a83eb79fec6ace1be66d51fd00fd4c966452a6b09521e7ea8aeb72b Jul 24 15:49:16.322687 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:16.322658 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" event={"ID":"3ae2059d-1ccb-4caf-a4c8-a110172b594b","Type":"ContainerStarted","Data":"f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef"} Jul 24 15:49:16.322861 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:16.322709 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" event={"ID":"3ae2059d-1ccb-4caf-a4c8-a110172b594b","Type":"ContainerStarted","Data":"7e5dcaf05a83eb79fec6ace1be66d51fd00fd4c966452a6b09521e7ea8aeb72b"} Jul 24 15:49:16.322909 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:16.322874 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:16.373622 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:16.373553 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" podStartSLOduration=1.373536003 podStartE2EDuration="1.373536003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:16.372331606 +0000 UTC m=+487.903919219" watchObservedRunningTime="2026-07-24 15:49:16.373536003 +0000 UTC m=+487.905123615" Jul 24 15:49:26.325134 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:26.325097 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:27.330933 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.330899 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 15:49:27.421636 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.421605 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j"] Jul 24 15:49:27.421846 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.421823 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" podUID="b4a660dd-5006-4f65-92bf-fddaf01d388f" containerName="manager" containerID="cri-o://6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae" gracePeriod=10 Jul 24 15:49:27.676729 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.676706 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:27.801349 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.801321 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a660dd-5006-4f65-92bf-fddaf01d388f-extensions-socket-volume\") pod \"b4a660dd-5006-4f65-92bf-fddaf01d388f\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " Jul 24 15:49:27.801512 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.801485 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8knk\" (UniqueName: \"kubernetes.io/projected/b4a660dd-5006-4f65-92bf-fddaf01d388f-kube-api-access-q8knk\") pod \"b4a660dd-5006-4f65-92bf-fddaf01d388f\" (UID: \"b4a660dd-5006-4f65-92bf-fddaf01d388f\") " Jul 24 15:49:27.801769 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.801746 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a660dd-5006-4f65-92bf-fddaf01d388f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b4a660dd-5006-4f65-92bf-fddaf01d388f" (UID: "b4a660dd-5006-4f65-92bf-fddaf01d388f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 15:49:27.803594 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.803540 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a660dd-5006-4f65-92bf-fddaf01d388f-kube-api-access-q8knk" (OuterVolumeSpecName: "kube-api-access-q8knk") pod "b4a660dd-5006-4f65-92bf-fddaf01d388f" (UID: "b4a660dd-5006-4f65-92bf-fddaf01d388f"). InnerVolumeSpecName "kube-api-access-q8knk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:49:27.902627 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.902533 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8knk\" (UniqueName: \"kubernetes.io/projected/b4a660dd-5006-4f65-92bf-fddaf01d388f-kube-api-access-q8knk\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:49:27.902627 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:27.902598 2543 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4a660dd-5006-4f65-92bf-fddaf01d388f-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:49:28.391048 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.391006 2543 generic.go:356] "Generic (PLEG): container finished" podID="b4a660dd-5006-4f65-92bf-fddaf01d388f" containerID="6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae" exitCode=0 Jul 24 15:49:28.391048 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.391052 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" Jul 24 15:49:28.391593 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.391085 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" event={"ID":"b4a660dd-5006-4f65-92bf-fddaf01d388f","Type":"ContainerDied","Data":"6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae"} Jul 24 15:49:28.391593 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.391135 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j" event={"ID":"b4a660dd-5006-4f65-92bf-fddaf01d388f","Type":"ContainerDied","Data":"70152b18aca99d4a5d6c942cc2e1af7eea1950f8a391f1863ed70020a2de75a1"} Jul 24 15:49:28.391593 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.391156 2543 scope.go:117] "RemoveContainer" containerID="6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae" Jul 24 15:49:28.400851 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.400834 2543 scope.go:117] "RemoveContainer" containerID="6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae" Jul 24 15:49:28.401109 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:49:28.401092 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae\": container with ID starting with 6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae not found: ID does not exist" containerID="6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae" Jul 24 15:49:28.401155 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.401118 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae"} err="failed to get container status \"6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae\": rpc error: code = NotFound desc = could not find container \"6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae\": container with ID starting with 6cb99e763e7827a5f081369e156d433cd7924ab3f86a5b9853135224641c8aae not found: ID does not exist" Jul 24 15:49:28.431212 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.431184 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j"] Jul 24 15:49:28.445480 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:28.445451 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-zdc7j"] Jul 24 15:49:29.043233 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:29.043202 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a660dd-5006-4f65-92bf-fddaf01d388f" path="/var/lib/kubelet/pods/b4a660dd-5006-4f65-92bf-fddaf01d388f/volumes" Jul 24 15:49:43.832764 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.832735 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk"] Jul 24 15:49:43.833229 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.833212 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4a660dd-5006-4f65-92bf-fddaf01d388f" containerName="manager" Jul 24 15:49:43.833229 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.833230 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4a660dd-5006-4f65-92bf-fddaf01d388f" containerName="manager" Jul 24 15:49:43.833374 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.833311 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4a660dd-5006-4f65-92bf-fddaf01d388f" containerName="manager" Jul 24 15:49:43.835289 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.835272 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.839240 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.839221 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-qvwzv\"" Jul 24 15:49:43.855779 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.855748 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk"] Jul 24 15:49:43.949068 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949037 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949217 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949083 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949217 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949145 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949217 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949211 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe0f77-6158-42c0-b519-77cfc6949af3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949319 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949239 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949319 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949264 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949319 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949287 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949319 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949306 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2568l\" (UniqueName: \"kubernetes.io/projected/54fe0f77-6158-42c0-b519-77cfc6949af3-kube-api-access-2568l\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:43.949441 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:43.949395 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050293 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050259 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050306 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050333 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050357 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe0f77-6158-42c0-b519-77cfc6949af3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050388 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050407 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050428 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050460 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2568l\" (UniqueName: \"kubernetes.io/projected/54fe0f77-6158-42c0-b519-77cfc6949af3-kube-api-access-2568l\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050864 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050500 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050864 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050752 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050864 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050786 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.050864 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050823 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.051072 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.050909 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.051216 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.051196 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe0f77-6158-42c0-b519-77cfc6949af3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.052958 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.052936 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.053141 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.053124 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.059033 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.059005 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe0f77-6158-42c0-b519-77cfc6949af3-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.059223 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.059203 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2568l\" (UniqueName: \"kubernetes.io/projected/54fe0f77-6158-42c0-b519-77cfc6949af3-kube-api-access-2568l\") pod \"maas-default-gateway-openshift-default-78bb78ff78-nj8hk\" (UID: \"54fe0f77-6158-42c0-b519-77cfc6949af3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.146358 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.146284 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:44.289812 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.289785 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk"] Jul 24 15:49:44.292419 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:49:44.292395 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fe0f77_6158_42c0_b519_77cfc6949af3.slice/crio-dd8f6be56f0aa03826aeed7edb6e0d8b28655a87240e359affe7e5a583b31d66 WatchSource:0}: Error finding container dd8f6be56f0aa03826aeed7edb6e0d8b28655a87240e359affe7e5a583b31d66: Status 404 returned error can't find the container with id dd8f6be56f0aa03826aeed7edb6e0d8b28655a87240e359affe7e5a583b31d66 Jul 24 15:49:44.294667 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.294629 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:49:44.294751 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.294714 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:49:44.294790 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.294754 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:49:44.471261 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.471216 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" event={"ID":"54fe0f77-6158-42c0-b519-77cfc6949af3","Type":"ContainerStarted","Data":"7926fb26c993c9e792cf563198e8e2bbe5d0932447e722a49ba519f1e7d36e5b"} Jul 24 15:49:44.471429 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.471282 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" event={"ID":"54fe0f77-6158-42c0-b519-77cfc6949af3","Type":"ContainerStarted","Data":"dd8f6be56f0aa03826aeed7edb6e0d8b28655a87240e359affe7e5a583b31d66"} Jul 24 15:49:44.492132 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:44.492086 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" podStartSLOduration=1.492070553 podStartE2EDuration="1.492070553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:44.489452518 +0000 UTC m=+516.021040180" watchObservedRunningTime="2026-07-24 15:49:44.492070553 +0000 UTC m=+516.023658166" Jul 24 15:49:45.146794 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.146767 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:45.150813 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.150788 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:45.476383 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.476353 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:45.477219 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.477204 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:49:45.520705 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.520675 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:49:45.524250 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.524227 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.526593 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.526573 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 15:49:45.526703 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.526618 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bsdpk\"" Jul 24 15:49:45.532490 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.532466 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:49:45.563220 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.563190 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c322fcb8-bcfc-445a-8bd0-4c4798883c68-config-file\") pod \"limitador-limitador-64c94fd64f-vfj9k\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.563338 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.563234 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwc67\" (UniqueName: \"kubernetes.io/projected/c322fcb8-bcfc-445a-8bd0-4c4798883c68-kube-api-access-wwc67\") pod \"limitador-limitador-64c94fd64f-vfj9k\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.616826 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.616795 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:49:45.664899 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.664869 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c322fcb8-bcfc-445a-8bd0-4c4798883c68-config-file\") pod \"limitador-limitador-64c94fd64f-vfj9k\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.665056 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.664911 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwc67\" (UniqueName: \"kubernetes.io/projected/c322fcb8-bcfc-445a-8bd0-4c4798883c68-kube-api-access-wwc67\") pod \"limitador-limitador-64c94fd64f-vfj9k\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.665599 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.665580 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c322fcb8-bcfc-445a-8bd0-4c4798883c68-config-file\") pod \"limitador-limitador-64c94fd64f-vfj9k\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.675411 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.675382 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwc67\" (UniqueName: \"kubernetes.io/projected/c322fcb8-bcfc-445a-8bd0-4c4798883c68-kube-api-access-wwc67\") pod \"limitador-limitador-64c94fd64f-vfj9k\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.836976 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.836883 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:45.977505 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:45.977482 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:49:45.979064 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:49:45.979024 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc322fcb8_bcfc_445a_8bd0_4c4798883c68.slice/crio-23fe107834f0a192d84a704784c4899bd51e37ac87284573ee47f09019b5d5b4 WatchSource:0}: Error finding container 23fe107834f0a192d84a704784c4899bd51e37ac87284573ee47f09019b5d5b4: Status 404 returned error can't find the container with id 23fe107834f0a192d84a704784c4899bd51e37ac87284573ee47f09019b5d5b4 Jul 24 15:49:46.483143 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:46.483104 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" event={"ID":"c322fcb8-bcfc-445a-8bd0-4c4798883c68","Type":"ContainerStarted","Data":"23fe107834f0a192d84a704784c4899bd51e37ac87284573ee47f09019b5d5b4"} Jul 24 15:49:49.502154 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:49.502116 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" event={"ID":"c322fcb8-bcfc-445a-8bd0-4c4798883c68","Type":"ContainerStarted","Data":"77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733"} Jul 24 15:49:49.502614 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:49.502262 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:49:49.520004 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:49:49.519960 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" podStartSLOduration=2.021026526 podStartE2EDuration="4.519945335s" totalImagesPullingTime="2.498918809s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:49:49.517723486 +0000 UTC m=+521.049311107" watchObservedRunningTime="2026-07-24 15:49:49.519945335 +0000 UTC m=+521.051532948" Jul 24 15:50:00.508801 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:00.508773 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:50:03.650849 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:03.650809 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:50:03.651316 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:03.651025 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" podUID="c322fcb8-bcfc-445a-8bd0-4c4798883c68" containerName="limitador" containerID="cri-o://77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733" gracePeriod=30 Jul 24 15:50:04.209062 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.209041 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:50:04.345899 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.345873 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wwc67\" (UniqueName: \"kubernetes.io/projected/c322fcb8-bcfc-445a-8bd0-4c4798883c68-kube-api-access-wwc67\") pod \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " Jul 24 15:50:04.346069 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.345942 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c322fcb8-bcfc-445a-8bd0-4c4798883c68-config-file\") pod \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\" (UID: \"c322fcb8-bcfc-445a-8bd0-4c4798883c68\") " Jul 24 15:50:04.346314 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.346288 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c322fcb8-bcfc-445a-8bd0-4c4798883c68-config-file" (OuterVolumeSpecName: "config-file") pod "c322fcb8-bcfc-445a-8bd0-4c4798883c68" (UID: "c322fcb8-bcfc-445a-8bd0-4c4798883c68"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 15:50:04.348076 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.348052 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c322fcb8-bcfc-445a-8bd0-4c4798883c68-kube-api-access-wwc67" (OuterVolumeSpecName: "kube-api-access-wwc67") pod "c322fcb8-bcfc-445a-8bd0-4c4798883c68" (UID: "c322fcb8-bcfc-445a-8bd0-4c4798883c68"). InnerVolumeSpecName "kube-api-access-wwc67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 15:50:04.447459 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.447432 2543 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c322fcb8-bcfc-445a-8bd0-4c4798883c68-config-file\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:50:04.447459 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.447455 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwc67\" (UniqueName: \"kubernetes.io/projected/c322fcb8-bcfc-445a-8bd0-4c4798883c68-kube-api-access-wwc67\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 15:50:04.577260 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.577222 2543 generic.go:356] "Generic (PLEG): container finished" podID="c322fcb8-bcfc-445a-8bd0-4c4798883c68" containerID="77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733" exitCode=0 Jul 24 15:50:04.577260 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.577262 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" Jul 24 15:50:04.577456 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.577309 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" event={"ID":"c322fcb8-bcfc-445a-8bd0-4c4798883c68","Type":"ContainerDied","Data":"77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733"} Jul 24 15:50:04.577456 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.577357 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-vfj9k" event={"ID":"c322fcb8-bcfc-445a-8bd0-4c4798883c68","Type":"ContainerDied","Data":"23fe107834f0a192d84a704784c4899bd51e37ac87284573ee47f09019b5d5b4"} Jul 24 15:50:04.577456 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.577383 2543 scope.go:117] "RemoveContainer" containerID="77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733" Jul 24 15:50:04.586385 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.586370 2543 scope.go:117] "RemoveContainer" containerID="77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733" Jul 24 15:50:04.586694 ip-10-0-143-210 kubenswrapper[2543]: E0724 15:50:04.586676 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733\": container with ID starting with 77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733 not found: ID does not exist" containerID="77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733" Jul 24 15:50:04.586749 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.586703 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733"} err="failed to get container status \"77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733\": rpc error: code = NotFound desc = could not find container \"77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733\": container with ID starting with 77a730ec20a357703d7b086bad8c85e4e433510f964a2a27ffad8841f7dd1733 not found: ID does not exist" Jul 24 15:50:04.597545 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.597490 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:50:04.601385 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:04.601361 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-vfj9k"] Jul 24 15:50:05.043790 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:05.043759 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c322fcb8-bcfc-445a-8bd0-4c4798883c68" path="/var/lib/kubelet/pods/c322fcb8-bcfc-445a-8bd0-4c4798883c68/volumes" Jul 24 15:50:07.304258 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.304218 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qswrs"] Jul 24 15:50:07.304903 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.304884 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c322fcb8-bcfc-445a-8bd0-4c4798883c68" containerName="limitador" Jul 24 15:50:07.304903 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.304903 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c322fcb8-bcfc-445a-8bd0-4c4798883c68" containerName="limitador" Jul 24 15:50:07.305076 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.305006 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c322fcb8-bcfc-445a-8bd0-4c4798883c68" containerName="limitador" Jul 24 15:50:07.309823 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.309799 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.312511 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.312478 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 15:50:07.312661 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.312511 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 15:50:07.312661 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.312546 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-cb9zd\"" Jul 24 15:50:07.313372 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.313359 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 24 15:50:07.317030 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.317009 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qswrs"] Jul 24 15:50:07.474843 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.474810 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a408116f-e35f-49ce-aac4-4deb1762e2b8-data\") pod \"postgres-69fb4594cf-qswrs\" (UID: \"a408116f-e35f-49ce-aac4-4deb1762e2b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.475005 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.474865 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx42s\" (UniqueName: \"kubernetes.io/projected/a408116f-e35f-49ce-aac4-4deb1762e2b8-kube-api-access-zx42s\") pod \"postgres-69fb4594cf-qswrs\" (UID: \"a408116f-e35f-49ce-aac4-4deb1762e2b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.576129 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.576050 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a408116f-e35f-49ce-aac4-4deb1762e2b8-data\") pod \"postgres-69fb4594cf-qswrs\" (UID: \"a408116f-e35f-49ce-aac4-4deb1762e2b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.576255 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.576127 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx42s\" (UniqueName: \"kubernetes.io/projected/a408116f-e35f-49ce-aac4-4deb1762e2b8-kube-api-access-zx42s\") pod \"postgres-69fb4594cf-qswrs\" (UID: \"a408116f-e35f-49ce-aac4-4deb1762e2b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.576491 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.576468 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a408116f-e35f-49ce-aac4-4deb1762e2b8-data\") pod \"postgres-69fb4594cf-qswrs\" (UID: \"a408116f-e35f-49ce-aac4-4deb1762e2b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.584919 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.584888 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx42s\" (UniqueName: \"kubernetes.io/projected/a408116f-e35f-49ce-aac4-4deb1762e2b8-kube-api-access-zx42s\") pod \"postgres-69fb4594cf-qswrs\" (UID: \"a408116f-e35f-49ce-aac4-4deb1762e2b8\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.623807 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.623788 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:07.756653 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:07.756626 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-qswrs"] Jul 24 15:50:07.758373 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:50:07.758349 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda408116f_e35f_49ce_aac4_4deb1762e2b8.slice/crio-2c5d6904a8d63702e57e6785e643fb071e29e048c86700e3d07e0c1c8b18c042 WatchSource:0}: Error finding container 2c5d6904a8d63702e57e6785e643fb071e29e048c86700e3d07e0c1c8b18c042: Status 404 returned error can't find the container with id 2c5d6904a8d63702e57e6785e643fb071e29e048c86700e3d07e0c1c8b18c042 Jul 24 15:50:08.606067 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:08.606030 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" event={"ID":"a408116f-e35f-49ce-aac4-4deb1762e2b8","Type":"ContainerStarted","Data":"2c5d6904a8d63702e57e6785e643fb071e29e048c86700e3d07e0c1c8b18c042"} Jul 24 15:50:12.565774 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:12.565751 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 24 15:50:13.639160 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:13.639122 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" event={"ID":"a408116f-e35f-49ce-aac4-4deb1762e2b8","Type":"ContainerStarted","Data":"09c1902caf56b095ac4e120277a7f240760240480341c6bb1319e25b8360ec40"} Jul 24 15:50:13.639588 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:13.639198 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:50:13.657159 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:13.657097 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" podStartSLOduration=1.85367651 podStartE2EDuration="6.65708246s" totalImagesPullingTime="4.80340595s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:50:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:50:13.655666671 +0000 UTC m=+545.187254310" watchObservedRunningTime="2026-07-24 15:50:13.65708246 +0000 UTC m=+545.188670120" Jul 24 15:50:19.673423 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:50:19.673395 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-qswrs" Jul 24 15:51:04.066849 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.066813 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8lj4t"] Jul 24 15:51:04.070299 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.070280 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.072933 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.072910 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 24 15:51:04.073082 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.073059 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ww5p6\"" Jul 24 15:51:04.080922 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.080898 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8lj4t"] Jul 24 15:51:04.116191 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.116155 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d58e57f-6052-4e74-9cf5-6114eb69024f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8lj4t\" (UID: \"9d58e57f-6052-4e74-9cf5-6114eb69024f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.116191 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.116189 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxd7h\" (UniqueName: \"kubernetes.io/projected/9d58e57f-6052-4e74-9cf5-6114eb69024f-kube-api-access-mxd7h\") pod \"payload-processing-5cc6cd444b-8lj4t\" (UID: \"9d58e57f-6052-4e74-9cf5-6114eb69024f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.217641 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.217606 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d58e57f-6052-4e74-9cf5-6114eb69024f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8lj4t\" (UID: \"9d58e57f-6052-4e74-9cf5-6114eb69024f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.217641 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.217642 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxd7h\" (UniqueName: \"kubernetes.io/projected/9d58e57f-6052-4e74-9cf5-6114eb69024f-kube-api-access-mxd7h\") pod \"payload-processing-5cc6cd444b-8lj4t\" (UID: \"9d58e57f-6052-4e74-9cf5-6114eb69024f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.218364 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.218338 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d58e57f-6052-4e74-9cf5-6114eb69024f-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-8lj4t\" (UID: \"9d58e57f-6052-4e74-9cf5-6114eb69024f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.228319 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.228296 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxd7h\" (UniqueName: \"kubernetes.io/projected/9d58e57f-6052-4e74-9cf5-6114eb69024f-kube-api-access-mxd7h\") pod \"payload-processing-5cc6cd444b-8lj4t\" (UID: \"9d58e57f-6052-4e74-9cf5-6114eb69024f\") " pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.383866 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.383779 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:04.521143 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.521112 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-8lj4t"] Jul 24 15:51:04.525385 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:51:04.525343 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d58e57f_6052_4e74_9cf5_6114eb69024f.slice/crio-9aa3ec5d8d030ed88d6b9bec144d16ad9550cd9e621dc873f1f4c147efdf7ec0 WatchSource:0}: Error finding container 9aa3ec5d8d030ed88d6b9bec144d16ad9550cd9e621dc873f1f4c147efdf7ec0: Status 404 returned error can't find the container with id 9aa3ec5d8d030ed88d6b9bec144d16ad9550cd9e621dc873f1f4c147efdf7ec0 Jul 24 15:51:04.889169 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:04.889134 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" event={"ID":"9d58e57f-6052-4e74-9cf5-6114eb69024f","Type":"ContainerStarted","Data":"9aa3ec5d8d030ed88d6b9bec144d16ad9550cd9e621dc873f1f4c147efdf7ec0"} Jul 24 15:51:06.900511 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:06.900481 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" event={"ID":"9d58e57f-6052-4e74-9cf5-6114eb69024f","Type":"ContainerStarted","Data":"7ea334370e467d0809628cb4ffbbf9b5bd5e3faa780606bbd964410d26413129"} Jul 24 15:51:06.901005 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:06.900598 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:51:06.919592 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:06.919516 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" podStartSLOduration=1.070066303 podStartE2EDuration="2.919484166s" totalImagesPullingTime="1.849417863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:51:06.915891051 +0000 UTC m=+598.447478686" watchObservedRunningTime="2026-07-24 15:51:06.919484166 +0000 UTC m=+598.451071781" Jul 24 15:51:11.941905 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:11.941811 2543 generic.go:356] "Generic (PLEG): container finished" podID="54fe0f77-6158-42c0-b519-77cfc6949af3" containerID="7926fb26c993c9e792cf563198e8e2bbe5d0932447e722a49ba519f1e7d36e5b" exitCode=0 Jul 24 15:51:11.941905 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:11.941887 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" event={"ID":"54fe0f77-6158-42c0-b519-77cfc6949af3","Type":"ContainerDied","Data":"7926fb26c993c9e792cf563198e8e2bbe5d0932447e722a49ba519f1e7d36e5b"} Jul 24 15:51:11.942340 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:11.942266 2543 scope.go:117] "RemoveContainer" containerID="7926fb26c993c9e792cf563198e8e2bbe5d0932447e722a49ba519f1e7d36e5b" Jul 24 15:51:11.943255 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:11.943227 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:51:11.943358 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:11.943308 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:51:11.943420 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:11.943353 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 15:51:12.948449 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:12.948420 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" event={"ID":"54fe0f77-6158-42c0-b519-77cfc6949af3","Type":"ContainerStarted","Data":"bb2ac66512277052470aa0502b6404057c884659e986962608daf3a21fd8a8a5"} Jul 24 15:51:13.147343 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:13.147303 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:51:13.148455 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:13.148425 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:51:13.148615 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:13.148492 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" podUID="54fe0f77-6158-42c0-b519-77cfc6949af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:51:14.147293 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:14.147263 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:51:14.147703 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:14.147317 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" podUID="54fe0f77-6158-42c0-b519-77cfc6949af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:51:15.146875 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:15.146835 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:51:15.147052 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:15.146894 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" podUID="54fe0f77-6158-42c0-b519-77cfc6949af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:51:15.476685 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:15.476659 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:51:16.147640 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:16.147607 2543 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-nj8hk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Jul 24 15:51:16.147824 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:16.147662 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" podUID="54fe0f77-6158-42c0-b519-77cfc6949af3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Jul 24 15:51:17.150918 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:17.150885 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:51:17.151984 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:17.151957 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-nj8hk" Jul 24 15:51:17.922752 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:51:17.922729 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-8lj4t" Jul 24 15:52:20.467580 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.467536 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5"] Jul 24 15:52:20.471347 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.471328 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.474528 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.474486 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-lmqq6\"" Jul 24 15:52:20.474528 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.474486 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 24 15:52:20.474802 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.474536 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 15:52:20.474802 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.474546 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 15:52:20.481481 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.481458 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5"] Jul 24 15:52:20.567317 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.567281 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.567489 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.567353 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.567489 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.567401 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.567489 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.567463 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.567665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.567492 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2647b3c-defc-4c44-a46e-9f591b62ba19-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.567665 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.567534 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg2fs\" (UniqueName: \"kubernetes.io/projected/f2647b3c-defc-4c44-a46e-9f591b62ba19-kube-api-access-hg2fs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.668960 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.668932 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg2fs\" (UniqueName: \"kubernetes.io/projected/f2647b3c-defc-4c44-a46e-9f591b62ba19-kube-api-access-hg2fs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.668982 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669032 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669051 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669138 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669083 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669322 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669217 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2647b3c-defc-4c44-a46e-9f591b62ba19-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669429 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669411 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669506 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669460 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.669602 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.669531 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.671477 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.671457 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f2647b3c-defc-4c44-a46e-9f591b62ba19-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.671808 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.671789 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f2647b3c-defc-4c44-a46e-9f591b62ba19-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.676738 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.676717 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg2fs\" (UniqueName: \"kubernetes.io/projected/f2647b3c-defc-4c44-a46e-9f591b62ba19-kube-api-access-hg2fs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tmdc5\" (UID: \"f2647b3c-defc-4c44-a46e-9f591b62ba19\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.784201 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.784169 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:20.921641 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.921615 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5"] Jul 24 15:52:20.923643 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:52:20.923618 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2647b3c_defc_4c44_a46e_9f591b62ba19.slice/crio-04c6c29e19d08c24a4436ad7a3f99818fe067ad6b8351e87014a2d6c800cf1b1 WatchSource:0}: Error finding container 04c6c29e19d08c24a4436ad7a3f99818fe067ad6b8351e87014a2d6c800cf1b1: Status 404 returned error can't find the container with id 04c6c29e19d08c24a4436ad7a3f99818fe067ad6b8351e87014a2d6c800cf1b1 Jul 24 15:52:20.925180 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:20.925161 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 15:52:21.291156 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:21.291118 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" event={"ID":"f2647b3c-defc-4c44-a46e-9f591b62ba19","Type":"ContainerStarted","Data":"04c6c29e19d08c24a4436ad7a3f99818fe067ad6b8351e87014a2d6c800cf1b1"} Jul 24 15:52:22.297811 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:22.297770 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" event={"ID":"f2647b3c-defc-4c44-a46e-9f591b62ba19","Type":"ContainerStarted","Data":"8efdff4b8d591320779c13afd306b08fd7b6570487e34839ea13b8a24f90a6cf"} Jul 24 15:52:22.315751 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:22.315703 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" podStartSLOduration=1.2938985619999999 podStartE2EDuration="2.31568666s" totalImagesPullingTime="1.021788098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:22.314397883 +0000 UTC m=+673.845985499" watchObservedRunningTime="2026-07-24 15:52:22.31568666 +0000 UTC m=+673.847274272" Jul 24 15:52:30.784722 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:30.784675 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:30.785158 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:30.784794 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:30.797233 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:30.797208 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:31.356468 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:31.356438 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tmdc5" Jul 24 15:52:33.165895 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.165862 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k"] Jul 24 15:52:33.170162 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.170139 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.172466 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.172444 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 15:52:33.179260 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.179238 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k"] Jul 24 15:52:33.291065 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.291028 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mvfp\" (UniqueName: \"kubernetes.io/projected/6b311d69-5b8e-4714-a15e-d4535d10574d-kube-api-access-9mvfp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.291226 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.291109 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b311d69-5b8e-4714-a15e-d4535d10574d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.291226 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.291145 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.291226 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.291189 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.291335 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.291243 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.291335 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.291286 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392190 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392159 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b311d69-5b8e-4714-a15e-d4535d10574d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392190 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392198 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392232 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392262 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392293 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392445 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392315 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mvfp\" (UniqueName: \"kubernetes.io/projected/6b311d69-5b8e-4714-a15e-d4535d10574d-kube-api-access-9mvfp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392695 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392674 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392695 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392685 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.392791 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.392744 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.394707 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.394688 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6b311d69-5b8e-4714-a15e-d4535d10574d-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.394953 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.394934 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6b311d69-5b8e-4714-a15e-d4535d10574d-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.400379 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.400358 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mvfp\" (UniqueName: \"kubernetes.io/projected/6b311d69-5b8e-4714-a15e-d4535d10574d-kube-api-access-9mvfp\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k\" (UID: \"6b311d69-5b8e-4714-a15e-d4535d10574d\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.481195 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.481167 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:33.621752 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:33.621726 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k"] Jul 24 15:52:33.623469 ip-10-0-143-210 kubenswrapper[2543]: W0724 15:52:33.623441 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b311d69_5b8e_4714_a15e_d4535d10574d.slice/crio-774dd33e49a9bbfc865150f1c006b7ef5586665b56f70724efc54d5da73fc40d WatchSource:0}: Error finding container 774dd33e49a9bbfc865150f1c006b7ef5586665b56f70724efc54d5da73fc40d: Status 404 returned error can't find the container with id 774dd33e49a9bbfc865150f1c006b7ef5586665b56f70724efc54d5da73fc40d Jul 24 15:52:34.362120 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:34.362088 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" event={"ID":"6b311d69-5b8e-4714-a15e-d4535d10574d","Type":"ContainerStarted","Data":"75c18677759a476770c56d8250f144150f0ce25584dd7071fa83d83410d777fd"} Jul 24 15:52:34.362490 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:34.362135 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" event={"ID":"6b311d69-5b8e-4714-a15e-d4535d10574d","Type":"ContainerStarted","Data":"774dd33e49a9bbfc865150f1c006b7ef5586665b56f70724efc54d5da73fc40d"} Jul 24 15:52:34.381868 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:34.381816 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" podStartSLOduration=1.160274889 podStartE2EDuration="1.381800915s" totalImagesPullingTime="221.526026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 15:52:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 15:52:34.379418843 +0000 UTC m=+685.911006468" watchObservedRunningTime="2026-07-24 15:52:34.381800915 +0000 UTC m=+685.913388527" Jul 24 15:52:43.482308 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:43.482278 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:43.482711 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:43.482335 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:43.494881 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:43.494857 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 15:52:44.425798 ip-10-0-143-210 kubenswrapper[2543]: I0724 15:52:44.425771 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k" Jul 24 16:00:00.140973 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.140937 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5"] Jul 24 16:00:00.144743 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.144727 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" Jul 24 16:00:00.147136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.147118 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\"" Jul 24 16:00:00.154101 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.154075 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" Jul 24 16:00:00.165026 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.165005 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5"] Jul 24 16:00:00.499983 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.499938 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5"] Jul 24 16:00:00.502511 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:00:00.502481 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85ad2343_4950_4d8f_89b6_17abe2cb063f.slice/crio-c5564eadec8ae6748ee54e8dcaae5b1faefabd8e0b215c0ec06c6ac662660a66 WatchSource:0}: Error finding container c5564eadec8ae6748ee54e8dcaae5b1faefabd8e0b215c0ec06c6ac662660a66: Status 404 returned error can't find the container with id c5564eadec8ae6748ee54e8dcaae5b1faefabd8e0b215c0ec06c6ac662660a66 Jul 24 16:00:00.504195 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.504180 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:00:00.688471 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:00.688437 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" event={"ID":"85ad2343-4950-4d8f-89b6-17abe2cb063f","Type":"ContainerStarted","Data":"c5564eadec8ae6748ee54e8dcaae5b1faefabd8e0b215c0ec06c6ac662660a66"} Jul 24 16:00:03.706359 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:03.706321 2543 generic.go:356] "Generic (PLEG): container finished" podID="85ad2343-4950-4d8f-89b6-17abe2cb063f" containerID="f9c9b720ce420bb86960333865892cbe12602d8409c3ca879d3d0dafc35f500d" exitCode=0 Jul 24 16:00:03.706794 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:03.706413 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" event={"ID":"85ad2343-4950-4d8f-89b6-17abe2cb063f","Type":"ContainerDied","Data":"f9c9b720ce420bb86960333865892cbe12602d8409c3ca879d3d0dafc35f500d"} Jul 24 16:00:04.855237 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:04.855211 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" Jul 24 16:00:05.718664 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:05.718634 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" Jul 24 16:00:05.718836 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:05.718635 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748480-nwsn5" event={"ID":"85ad2343-4950-4d8f-89b6-17abe2cb063f","Type":"ContainerDied","Data":"c5564eadec8ae6748ee54e8dcaae5b1faefabd8e0b215c0ec06c6ac662660a66"} Jul 24 16:00:05.718836 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:00:05.718758 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5564eadec8ae6748ee54e8dcaae5b1faefabd8e0b215c0ec06c6ac662660a66" Jul 24 16:01:08.903907 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:08.903865 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj"] Jul 24 16:01:08.904493 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:08.904072 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" podUID="3ae2059d-1ccb-4caf-a4c8-a110172b594b" containerName="manager" containerID="cri-o://f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef" gracePeriod=10 Jul 24 16:01:12.066716 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.066647 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 16:01:12.071080 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.071040 2543 generic.go:356] "Generic (PLEG): container finished" podID="3ae2059d-1ccb-4caf-a4c8-a110172b594b" containerID="f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef" exitCode=0 Jul 24 16:01:12.071203 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.071084 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" event={"ID":"3ae2059d-1ccb-4caf-a4c8-a110172b594b","Type":"ContainerDied","Data":"f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef"} Jul 24 16:01:12.071203 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.071088 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" Jul 24 16:01:12.071203 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.071125 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj" event={"ID":"3ae2059d-1ccb-4caf-a4c8-a110172b594b","Type":"ContainerDied","Data":"7e5dcaf05a83eb79fec6ace1be66d51fd00fd4c966452a6b09521e7ea8aeb72b"} Jul 24 16:01:12.071203 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.071147 2543 scope.go:117] "RemoveContainer" containerID="f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef" Jul 24 16:01:12.080435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.080310 2543 scope.go:117] "RemoveContainer" containerID="f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef" Jul 24 16:01:12.080622 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:01:12.080601 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef\": container with ID starting with f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef not found: ID does not exist" containerID="f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef" Jul 24 16:01:12.080715 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.080631 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef"} err="failed to get container status \"f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef\": rpc error: code = NotFound desc = could not find container \"f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef\": container with ID starting with f28dbbe58c839954c3b7c73ffb70b69d874ffb77befd89a672190bb16f00e7ef not found: ID does not exist" Jul 24 16:01:12.220655 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.220626 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae2059d-1ccb-4caf-a4c8-a110172b594b-extensions-socket-volume\") pod \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " Jul 24 16:01:12.220829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.220704 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bpbnq\" (UniqueName: \"kubernetes.io/projected/3ae2059d-1ccb-4caf-a4c8-a110172b594b-kube-api-access-bpbnq\") pod \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\" (UID: \"3ae2059d-1ccb-4caf-a4c8-a110172b594b\") " Jul 24 16:01:12.220972 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.220952 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae2059d-1ccb-4caf-a4c8-a110172b594b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "3ae2059d-1ccb-4caf-a4c8-a110172b594b" (UID: "3ae2059d-1ccb-4caf-a4c8-a110172b594b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:01:12.222859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.222836 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae2059d-1ccb-4caf-a4c8-a110172b594b-kube-api-access-bpbnq" (OuterVolumeSpecName: "kube-api-access-bpbnq") pod "3ae2059d-1ccb-4caf-a4c8-a110172b594b" (UID: "3ae2059d-1ccb-4caf-a4c8-a110172b594b"). InnerVolumeSpecName "kube-api-access-bpbnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:01:12.321897 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.321828 2543 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae2059d-1ccb-4caf-a4c8-a110172b594b-extensions-socket-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:01:12.321897 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.321852 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpbnq\" (UniqueName: \"kubernetes.io/projected/3ae2059d-1ccb-4caf-a4c8-a110172b594b-kube-api-access-bpbnq\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:01:12.393890 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.393861 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj"] Jul 24 16:01:12.399055 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:12.399033 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w87cj"] Jul 24 16:01:13.043340 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:01:13.043307 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ae2059d-1ccb-4caf-a4c8-a110172b594b" path="/var/lib/kubelet/pods/3ae2059d-1ccb-4caf-a4c8-a110172b594b/volumes" Jul 24 16:02:10.070674 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.070638 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4"] Jul 24 16:02:10.071079 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.071054 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae2059d-1ccb-4caf-a4c8-a110172b594b" containerName="manager" Jul 24 16:02:10.071079 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.071065 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae2059d-1ccb-4caf-a4c8-a110172b594b" containerName="manager" Jul 24 16:02:10.071151 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.071088 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85ad2343-4950-4d8f-89b6-17abe2cb063f" containerName="cleanup" Jul 24 16:02:10.071151 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.071094 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="85ad2343-4950-4d8f-89b6-17abe2cb063f" containerName="cleanup" Jul 24 16:02:10.071151 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.071147 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae2059d-1ccb-4caf-a4c8-a110172b594b" containerName="manager" Jul 24 16:02:10.071246 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.071160 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="85ad2343-4950-4d8f-89b6-17abe2cb063f" containerName="cleanup" Jul 24 16:02:10.075212 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.075195 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.077687 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.077664 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wq5kn\"" Jul 24 16:02:10.087640 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.087619 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4"] Jul 24 16:02:10.184068 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.184039 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr857\" (UniqueName: \"kubernetes.io/projected/80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2-kube-api-access-hr857\") pod \"kuadrant-operator-controller-manager-86878c769f-njcn4\" (UID: \"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.184254 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.184081 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-njcn4\" (UID: \"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.284954 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.284922 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-njcn4\" (UID: \"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.285102 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.285088 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr857\" (UniqueName: \"kubernetes.io/projected/80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2-kube-api-access-hr857\") pod \"kuadrant-operator-controller-manager-86878c769f-njcn4\" (UID: \"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.285315 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.285298 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-njcn4\" (UID: \"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.295999 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.295976 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr857\" (UniqueName: \"kubernetes.io/projected/80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2-kube-api-access-hr857\") pod \"kuadrant-operator-controller-manager-86878c769f-njcn4\" (UID: \"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.387935 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.387862 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:10.527323 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:10.527299 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4"] Jul 24 16:02:10.529341 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:02:10.529317 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cd8ae4_ead0_4fe8_9f2f_0abafdc169f2.slice/crio-36eb45082b33925495c288b682cebd7904bff4f2d6229d828094ec0d2d30a5fd WatchSource:0}: Error finding container 36eb45082b33925495c288b682cebd7904bff4f2d6229d828094ec0d2d30a5fd: Status 404 returned error can't find the container with id 36eb45082b33925495c288b682cebd7904bff4f2d6229d828094ec0d2d30a5fd Jul 24 16:02:11.370375 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:11.370342 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" event={"ID":"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2","Type":"ContainerStarted","Data":"8e1ee114d6d6f9712742ec71a6ac529b686a4ae9c5d0f2f0d8fc40e7cb2832b2"} Jul 24 16:02:11.370937 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:11.370391 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" event={"ID":"80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2","Type":"ContainerStarted","Data":"36eb45082b33925495c288b682cebd7904bff4f2d6229d828094ec0d2d30a5fd"} Jul 24 16:02:11.370937 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:11.370464 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:02:11.390500 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:11.390444 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" podStartSLOduration=1.390419232 podStartE2EDuration="1.390419232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:02:11.388971812 +0000 UTC m=+1262.920559452" watchObservedRunningTime="2026-07-24 16:02:11.390419232 +0000 UTC m=+1262.922006849" Jul 24 16:02:22.377801 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:02:22.377762 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-njcn4" Jul 24 16:09:06.895817 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.895787 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967"] Jul 24 16:09:06.899518 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.899501 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.901952 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.901934 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a42991b8-openshift-default-dockercfg-28vk4\"" Jul 24 16:09:06.918583 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.918531 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967"] Jul 24 16:09:06.986105 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986069 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-credential-socket\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986290 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986126 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-podinfo\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986290 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986150 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-envoy\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986290 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986170 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jgft\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-kube-api-access-7jgft\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986290 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986203 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-token\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986290 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986261 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istiod-ca-cert\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986531 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986295 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-socket\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986531 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986324 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-certs\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:06.986531 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:06.986375 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-data\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087173 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087140 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-socket\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087184 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-certs\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087199 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-data\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087244 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-credential-socket\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087295 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-podinfo\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087329 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-envoy\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087668 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087355 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jgft\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-kube-api-access-7jgft\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087668 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087387 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-token\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087668 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087425 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istiod-ca-cert\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087668 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087608 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-socket\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087915 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087885 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-data\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.087974 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087931 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-credential-socket\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.088016 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.087988 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-certs\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.088181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.088166 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istiod-ca-cert\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.089730 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.089709 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-envoy\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.090004 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.089986 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-podinfo\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.100238 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.100205 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jgft\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-kube-api-access-7jgft\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.100338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.100301 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-token\") pod \"e2e-ait-a42991b8-openshift-default-5ff667984d-kd967\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.211374 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.211293 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:07.353158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.353135 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967"] Jul 24 16:09:07.354713 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:09:07.354690 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode48f6f4f_83a5_4f53_bcda_c4f0006e7ff8.slice/crio-20d0b9642d692194935a05ddeb35be7e2c30f575382f888f6ab3325a878a39e9 WatchSource:0}: Error finding container 20d0b9642d692194935a05ddeb35be7e2c30f575382f888f6ab3325a878a39e9: Status 404 returned error can't find the container with id 20d0b9642d692194935a05ddeb35be7e2c30f575382f888f6ab3325a878a39e9 Jul 24 16:09:07.356846 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.356817 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:07.356924 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.356879 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:07.356924 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.356906 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:07.527784 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.527751 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" event={"ID":"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8","Type":"ContainerStarted","Data":"97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f"} Jul 24 16:09:07.527958 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.527805 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" event={"ID":"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8","Type":"ContainerStarted","Data":"20d0b9642d692194935a05ddeb35be7e2c30f575382f888f6ab3325a878a39e9"} Jul 24 16:09:07.552148 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:07.552083 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podStartSLOduration=1.552039192 podStartE2EDuration="1.552039192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:09:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:09:07.55162618 +0000 UTC m=+1679.083213815" watchObservedRunningTime="2026-07-24 16:09:07.552039192 +0000 UTC m=+1679.083626805" Jul 24 16:09:08.211526 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:08.211495 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:08.216841 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:08.216815 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:08.533424 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:08.533393 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:08.534266 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:08.534249 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:24.002886 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.002191 2543 patch_prober.go:28] interesting pod/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:42216->10.134.0.44:15021: read: connection reset by peer" start-of-body= Jul 24 16:09:24.002886 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.002271 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": read tcp 10.134.0.2:42216->10.134.0.44:15021: read: connection reset by peer" Jul 24 16:09:24.620546 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.620515 2543 generic.go:356] "Generic (PLEG): container finished" podID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerID="97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f" exitCode=0 Jul 24 16:09:24.620736 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.620596 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" event={"ID":"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8","Type":"ContainerDied","Data":"97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f"} Jul 24 16:09:24.621005 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.620991 2543 scope.go:117] "RemoveContainer" containerID="97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f" Jul 24 16:09:24.622159 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.622133 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:24.622229 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.622213 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:24.622270 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:24.622255 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:25.645280 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:25.645241 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" event={"ID":"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8","Type":"ContainerStarted","Data":"46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429"} Jul 24 16:09:26.212050 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:26.212010 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:26.212050 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:26.212045 2543 patch_prober.go:28] interesting pod/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 24 16:09:26.212325 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:26.212102 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 24 16:09:27.212688 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:27.212646 2543 patch_prober.go:28] interesting pod/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 24 16:09:27.213201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:27.212716 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 24 16:09:28.212802 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:28.212763 2543 patch_prober.go:28] interesting pod/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 24 16:09:28.213380 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:28.213346 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 24 16:09:29.212931 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:29.212413 2543 patch_prober.go:28] interesting pod/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 24 16:09:29.212931 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:29.212477 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 24 16:09:30.215047 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:30.215017 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:30.215496 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:30.215282 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:30.216087 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:30.216073 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:34.495228 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.495197 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz"] Jul 24 16:09:34.497912 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.497896 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:34.500273 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.500246 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\"" Jul 24 16:09:34.507025 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.507003 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz"] Jul 24 16:09:34.574460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.574420 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65596169-6113-4531-a048-94d63bcc0cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz\" (UID: \"65596169-6113-4531-a048-94d63bcc0cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:34.675260 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.675228 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65596169-6113-4531-a048-94d63bcc0cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz\" (UID: \"65596169-6113-4531-a048-94d63bcc0cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:34.675834 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.675815 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65596169-6113-4531-a048-94d63bcc0cdf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz\" (UID: \"65596169-6113-4531-a048-94d63bcc0cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:34.809766 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.809668 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:34.945898 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:34.945871 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz"] Jul 24 16:09:34.947423 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:09:34.947395 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65596169_6113_4531_a048_94d63bcc0cdf.slice/crio-ff958aac96ea69b9048e2ec468e2911d65159c0ac354cffbcd92a05c1e5966dc WatchSource:0}: Error finding container ff958aac96ea69b9048e2ec468e2911d65159c0ac354cffbcd92a05c1e5966dc: Status 404 returned error can't find the container with id ff958aac96ea69b9048e2ec468e2911d65159c0ac354cffbcd92a05c1e5966dc Jul 24 16:09:35.701391 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:35.701363 2543 generic.go:356] "Generic (PLEG): container finished" podID="65596169-6113-4531-a048-94d63bcc0cdf" containerID="b458b8ad36b51626cc95eb6ba8f76b037a0ece179bb211aedd5ca9aed345a760" exitCode=0 Jul 24 16:09:35.701783 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:35.701487 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" event={"ID":"65596169-6113-4531-a048-94d63bcc0cdf","Type":"ContainerDied","Data":"b458b8ad36b51626cc95eb6ba8f76b037a0ece179bb211aedd5ca9aed345a760"} Jul 24 16:09:35.701783 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:35.701534 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" event={"ID":"65596169-6113-4531-a048-94d63bcc0cdf","Type":"ContainerStarted","Data":"ff958aac96ea69b9048e2ec468e2911d65159c0ac354cffbcd92a05c1e5966dc"} Jul 24 16:09:36.844323 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:36.844303 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:36.896119 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:36.896092 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65596169-6113-4531-a048-94d63bcc0cdf-maas-api-service-ca\") pod \"65596169-6113-4531-a048-94d63bcc0cdf\" (UID: \"65596169-6113-4531-a048-94d63bcc0cdf\") " Jul 24 16:09:36.896441 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:36.896417 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65596169-6113-4531-a048-94d63bcc0cdf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "65596169-6113-4531-a048-94d63bcc0cdf" (UID: "65596169-6113-4531-a048-94d63bcc0cdf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:36.997260 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:36.997223 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65596169-6113-4531-a048-94d63bcc0cdf-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:37.712701 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:37.712669 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" Jul 24 16:09:37.712875 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:37.712716 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz" event={"ID":"65596169-6113-4531-a048-94d63bcc0cdf","Type":"ContainerDied","Data":"ff958aac96ea69b9048e2ec468e2911d65159c0ac354cffbcd92a05c1e5966dc"} Jul 24 16:09:37.712875 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:37.712758 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff958aac96ea69b9048e2ec468e2911d65159c0ac354cffbcd92a05c1e5966dc" Jul 24 16:09:49.571250 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:49.571215 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz"] Jul 24 16:09:49.575413 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:49.575386 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a42991b8-fwdjz"] Jul 24 16:09:49.706545 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:49.706514 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967"] Jul 24 16:09:49.707092 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:49.707058 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" containerID="cri-o://46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429" gracePeriod=30 Jul 24 16:09:50.763398 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.763368 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm"] Jul 24 16:09:50.764008 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.763987 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65596169-6113-4531-a048-94d63bcc0cdf" containerName="revoke-keys" Jul 24 16:09:50.764101 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.764010 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="65596169-6113-4531-a048-94d63bcc0cdf" containerName="revoke-keys" Jul 24 16:09:50.764178 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.764164 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="65596169-6113-4531-a048-94d63bcc0cdf" containerName="revoke-keys" Jul 24 16:09:50.767134 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.767115 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.769735 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.769718 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-54f72705-gw-openshift-default-dockercfg-g8vt4\"" Jul 24 16:09:50.787883 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.787852 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm"] Jul 24 16:09:50.838441 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838411 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-credential-socket\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838599 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838448 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-data\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838599 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838474 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm7ns\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-kube-api-access-nm7ns\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838599 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838538 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-socket\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838599 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838597 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-envoy\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838757 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838624 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-token\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838757 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838681 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-certs\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838757 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838722 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istiod-ca-cert\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.838853 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.838780 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-podinfo\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939303 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-podinfo\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939358 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-credential-socket\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939386 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-data\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939413 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm7ns\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-kube-api-access-nm7ns\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939452 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-socket\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939477 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-envoy\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939507 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-token\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939549 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-certs\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939620 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istiod-ca-cert\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.939859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939786 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-credential-socket\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.940025 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.939899 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-socket\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.940109 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.940072 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-certs\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.940279 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.940260 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-data\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.940544 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.940306 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istiod-ca-cert\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.942004 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.941975 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-envoy\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.942188 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.942172 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-podinfo\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.947728 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.947710 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-token\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:50.948010 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:50.947992 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm7ns\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-kube-api-access-nm7ns\") pod \"e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:51.043402 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.043340 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65596169-6113-4531-a048-94d63bcc0cdf" path="/var/lib/kubelet/pods/65596169-6113-4531-a048-94d63bcc0cdf/volumes" Jul 24 16:09:51.078177 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.078147 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:51.221873 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.221846 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm"] Jul 24 16:09:51.223632 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:09:51.223604 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod043b6995_4ca5_4dbe_af1c_c3c73b287cfb.slice/crio-0cbc44a5013e17d1008681339222d9e7e4d7760179a8ce5f20663eeb8592ecdd WatchSource:0}: Error finding container 0cbc44a5013e17d1008681339222d9e7e4d7760179a8ce5f20663eeb8592ecdd: Status 404 returned error can't find the container with id 0cbc44a5013e17d1008681339222d9e7e4d7760179a8ce5f20663eeb8592ecdd Jul 24 16:09:51.225965 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.225928 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:51.226039 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.226016 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:51.226082 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.226059 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:09:51.784274 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.784238 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" event={"ID":"043b6995-4ca5-4dbe-af1c-c3c73b287cfb","Type":"ContainerStarted","Data":"7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00"} Jul 24 16:09:51.784676 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.784316 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" event={"ID":"043b6995-4ca5-4dbe-af1c-c3c73b287cfb","Type":"ContainerStarted","Data":"0cbc44a5013e17d1008681339222d9e7e4d7760179a8ce5f20663eeb8592ecdd"} Jul 24 16:09:51.808642 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:51.808591 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podStartSLOduration=1.8085751239999999 podStartE2EDuration="1.808575124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:09:51.805132925 +0000 UTC m=+1723.336720538" watchObservedRunningTime="2026-07-24 16:09:51.808575124 +0000 UTC m=+1723.340162729" Jul 24 16:09:52.078903 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:52.078816 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:52.083701 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:52.083680 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:52.789053 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:52.789024 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:52.790078 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:52.790060 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:09:54.970907 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:54.970888 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:55.092280 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092242 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-data\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092451 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092292 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-podinfo\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092451 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092313 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-socket\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092451 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092338 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-credential-socket\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092451 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092391 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-certs\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092451 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092426 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istiod-ca-cert\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092451 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092451 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-envoy\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092786 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092516 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-token\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092786 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092595 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jgft\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-kube-api-access-7jgft\") pod \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\" (UID: \"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8\") " Jul 24 16:09:55.092786 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092717 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:55.092786 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092770 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:55.092984 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.092955 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:55.093171 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.093119 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.093171 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.093170 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.093341 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.093184 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.093427 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.093404 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-data" (OuterVolumeSpecName: "istio-data") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:55.093727 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.093692 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:09:55.094876 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.094852 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:09:55.095169 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.095142 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-kube-api-access-7jgft" (OuterVolumeSpecName: "kube-api-access-7jgft") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "kube-api-access-7jgft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:55.095263 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.095184 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-token" (OuterVolumeSpecName: "istio-token") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:09:55.095400 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.095376 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" (UID: "e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:09:55.194106 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.194072 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.194106 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.194102 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.194338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.194115 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.194338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.194128 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.194338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.194139 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.194338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.194149 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jgft\" (UniqueName: \"kubernetes.io/projected/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8-kube-api-access-7jgft\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:09:55.806878 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.806843 2543 generic.go:356] "Generic (PLEG): container finished" podID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerID="46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429" exitCode=0 Jul 24 16:09:55.807089 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.806895 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" Jul 24 16:09:55.807089 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.806924 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" event={"ID":"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8","Type":"ContainerDied","Data":"46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429"} Jul 24 16:09:55.807089 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.806983 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967" event={"ID":"e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8","Type":"ContainerDied","Data":"20d0b9642d692194935a05ddeb35be7e2c30f575382f888f6ab3325a878a39e9"} Jul 24 16:09:55.807089 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.807014 2543 scope.go:117] "RemoveContainer" containerID="46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429" Jul 24 16:09:55.817429 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.817408 2543 scope.go:117] "RemoveContainer" containerID="97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f" Jul 24 16:09:55.825640 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.825623 2543 scope.go:117] "RemoveContainer" containerID="46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429" Jul 24 16:09:55.825931 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:09:55.825907 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429\": container with ID starting with 46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429 not found: ID does not exist" containerID="46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429" Jul 24 16:09:55.826006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.825944 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429"} err="failed to get container status \"46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429\": rpc error: code = NotFound desc = could not find container \"46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429\": container with ID starting with 46eb4601e7199d51e93bb4f28af8505e34ec5ac1aae8c09e168df60e8a138429 not found: ID does not exist" Jul 24 16:09:55.826006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.825968 2543 scope.go:117] "RemoveContainer" containerID="97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f" Jul 24 16:09:55.826219 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:09:55.826205 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f\": container with ID starting with 97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f not found: ID does not exist" containerID="97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f" Jul 24 16:09:55.826262 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.826223 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f"} err="failed to get container status \"97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f\": rpc error: code = NotFound desc = could not find container \"97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f\": container with ID starting with 97a2ceb56cf5dc1c2ac809b964a1b4cbd5180084aa17ccfff1617a14d653589f not found: ID does not exist" Jul 24 16:09:55.830153 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.830132 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967"] Jul 24 16:09:55.834595 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:55.834574 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a42991b8-openshift-default-5ff667984d-kd967"] Jul 24 16:09:57.044330 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:09:57.044289 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" path="/var/lib/kubelet/pods/e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8/volumes" Jul 24 16:10:07.790649 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.790608 2543 patch_prober.go:28] interesting pod/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 16:10:07.791068 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.790682 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 16:10:07.881646 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.881617 2543 generic.go:356] "Generic (PLEG): container finished" podID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerID="7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00" exitCode=0 Jul 24 16:10:07.881822 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.881676 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" event={"ID":"043b6995-4ca5-4dbe-af1c-c3c73b287cfb","Type":"ContainerDied","Data":"7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00"} Jul 24 16:10:07.882117 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.882094 2543 scope.go:117] "RemoveContainer" containerID="7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00" Jul 24 16:10:07.883192 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.883161 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:07.883259 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.883244 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:07.883308 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:07.883283 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:08.888874 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:08.888842 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" event={"ID":"043b6995-4ca5-4dbe-af1c-c3c73b287cfb","Type":"ContainerStarted","Data":"8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73"} Jul 24 16:10:09.078705 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:09.078277 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:10:09.078705 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:09.078548 2543 patch_prober.go:28] interesting pod/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 16:10:09.078705 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:09.078629 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 16:10:10.079516 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:10.079479 2543 patch_prober.go:28] interesting pod/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 16:10:10.080020 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:10.079532 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 16:10:11.078645 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:11.078608 2543 patch_prober.go:28] interesting pod/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 16:10:11.078827 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:11.078664 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 16:10:12.078944 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:12.078910 2543 patch_prober.go:28] interesting pod/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Jul 24 16:10:12.079349 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:12.078968 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Jul 24 16:10:13.082918 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:13.082892 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:10:13.083300 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:13.083216 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:10:13.083993 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:13.083971 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:10:18.294046 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294012 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc"] Jul 24 16:10:18.294475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294461 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" Jul 24 16:10:18.294475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294476 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" Jul 24 16:10:18.294554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294485 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" Jul 24 16:10:18.294554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294490 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" Jul 24 16:10:18.294554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294554 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" Jul 24 16:10:18.294554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.294585 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e48f6f4f-83a5-4f53-bcda-c4f0006e7ff8" containerName="istio-proxy" Jul 24 16:10:18.297343 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.297328 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:18.299792 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.299771 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\"" Jul 24 16:10:18.305371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.305351 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc"] Jul 24 16:10:18.420535 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.420497 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc\" (UID: \"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:18.521836 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.521801 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc\" (UID: \"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:18.522409 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.522392 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc\" (UID: \"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:18.609675 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.609584 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:18.952007 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:18.951979 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc"] Jul 24 16:10:18.953986 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:10:18.953960 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3fedb7e_55c9_43ea_8f11_ef3e3cebe3f3.slice/crio-8adf772e40b3798f6d80630d155e4325a5c476b32f6f386adf5849fbde769d4a WatchSource:0}: Error finding container 8adf772e40b3798f6d80630d155e4325a5c476b32f6f386adf5849fbde769d4a: Status 404 returned error can't find the container with id 8adf772e40b3798f6d80630d155e4325a5c476b32f6f386adf5849fbde769d4a Jul 24 16:10:19.947658 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:19.947556 2543 generic.go:356] "Generic (PLEG): container finished" podID="c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3" containerID="fe19961e61bcc0fc206ba99525b4983cc1d315c40072b3c4f26d920508d7678b" exitCode=0 Jul 24 16:10:19.947658 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:19.947625 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" event={"ID":"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3","Type":"ContainerDied","Data":"fe19961e61bcc0fc206ba99525b4983cc1d315c40072b3c4f26d920508d7678b"} Jul 24 16:10:19.948060 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:19.947678 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" event={"ID":"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3","Type":"ContainerStarted","Data":"8adf772e40b3798f6d80630d155e4325a5c476b32f6f386adf5849fbde769d4a"} Jul 24 16:10:21.092577 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.092546 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:21.148680 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.148648 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3-maas-api-service-ca\") pod \"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3\" (UID: \"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3\") " Jul 24 16:10:21.149010 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.148988 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3" (UID: "c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:10:21.249792 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.249758 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:21.958769 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.958735 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" Jul 24 16:10:21.958951 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.958735 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc" event={"ID":"c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3","Type":"ContainerDied","Data":"8adf772e40b3798f6d80630d155e4325a5c476b32f6f386adf5849fbde769d4a"} Jul 24 16:10:21.958951 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:21.958854 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8adf772e40b3798f6d80630d155e4325a5c476b32f6f386adf5849fbde769d4a" Jul 24 16:10:33.377792 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:33.377752 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc"] Jul 24 16:10:33.379906 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:33.379881 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54f72705-lhvlc"] Jul 24 16:10:33.516866 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:33.516838 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm"] Jul 24 16:10:33.517115 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:33.517073 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" containerID="cri-o://8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73" gracePeriod=30 Jul 24 16:10:34.322931 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.322902 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz"] Jul 24 16:10:34.323430 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.323413 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3" containerName="revoke-keys" Jul 24 16:10:34.323531 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.323433 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3" containerName="revoke-keys" Jul 24 16:10:34.323621 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.323604 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3" containerName="revoke-keys" Jul 24 16:10:34.326947 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.326927 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.329472 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.329449 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e125f3d6-openshift-default-dockercfg-dv24x\"" Jul 24 16:10:34.339517 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.339492 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz"] Jul 24 16:10:34.480955 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.480925 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-data\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.480967 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-socket\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481065 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-token\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481092 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrz87\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-kube-api-access-lrz87\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481117 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istiod-ca-cert\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481176 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-podinfo\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481211 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-certs\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481229 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-envoy\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.481334 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.481258 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-credential-socket\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.582907 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.582823 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-socket\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.582907 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.582897 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-token\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.582917 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrz87\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-kube-api-access-lrz87\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.582934 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istiod-ca-cert\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.582972 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-podinfo\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583004 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-certs\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583030 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-envoy\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583061 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-credential-socket\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583117 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-data\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583485 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583263 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-socket\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583485 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583458 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-certs\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583635 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583583 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-data\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583635 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583611 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-credential-socket\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.583805 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.583785 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istiod-ca-cert\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.585504 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.585487 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-envoy\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.585833 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.585817 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-podinfo\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.591162 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.591141 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-token\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.591285 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.591268 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrz87\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-kube-api-access-lrz87\") pod \"e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.638959 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.638923 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:34.784366 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.784335 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz"] Jul 24 16:10:34.790513 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.790475 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:34.790627 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.790586 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:34.790689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:34.790637 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:35.033297 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:35.033260 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" event={"ID":"cf6cd4f9-0de6-4990-b20b-865a5bb56133","Type":"ContainerStarted","Data":"3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15"} Jul 24 16:10:35.033488 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:35.033326 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" event={"ID":"cf6cd4f9-0de6-4990-b20b-865a5bb56133","Type":"ContainerStarted","Data":"b9fb432f49c745e912c9ba07050aab3ffd934fed5b64ae2878a0e9b1ff02e0d6"} Jul 24 16:10:35.044111 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:35.044074 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3" path="/var/lib/kubelet/pods/c3fedb7e-55c9-43ea-8f11-ef3e3cebe3f3/volumes" Jul 24 16:10:35.055261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:35.055183 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podStartSLOduration=1.055161812 podStartE2EDuration="1.055161812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:10:35.054206635 +0000 UTC m=+1766.585794249" watchObservedRunningTime="2026-07-24 16:10:35.055161812 +0000 UTC m=+1766.586749426" Jul 24 16:10:35.639764 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:35.639727 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:35.644818 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:35.644788 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:36.038822 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:36.038795 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:36.039687 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:36.039660 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:38.783475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.783415 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:10:38.927591 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927537 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nm7ns\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-kube-api-access-nm7ns\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.927808 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927615 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istiod-ca-cert\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.927808 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927644 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-podinfo\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.927808 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927692 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-data\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.927808 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927720 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-credential-socket\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.927808 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927784 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-socket\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.928065 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927812 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-envoy\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.928065 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927849 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-certs\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.928065 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.927887 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-token\") pod \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\" (UID: \"043b6995-4ca5-4dbe-af1c-c3c73b287cfb\") " Jul 24 16:10:38.928065 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928019 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:10:38.928251 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928079 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:10:38.928323 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928289 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:10:38.928532 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928462 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:38.928532 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928483 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:38.928532 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928497 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:38.928532 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928500 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:10:38.929052 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.928726 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-data" (OuterVolumeSpecName: "istio-data") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:10:38.930133 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.930109 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:10:38.930292 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.930261 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:10:38.930352 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.930317 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-kube-api-access-nm7ns" (OuterVolumeSpecName: "kube-api-access-nm7ns") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "kube-api-access-nm7ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:10:38.930554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:38.930540 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-token" (OuterVolumeSpecName: "istio-token") pod "043b6995-4ca5-4dbe-af1c-c3c73b287cfb" (UID: "043b6995-4ca5-4dbe-af1c-c3c73b287cfb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:10:39.029614 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.029548 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:39.029614 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.029609 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:39.029614 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.029620 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:39.029850 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.029629 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:39.029850 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.029646 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm7ns\" (UniqueName: \"kubernetes.io/projected/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-kube-api-access-nm7ns\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:39.029850 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.029654 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/043b6995-4ca5-4dbe-af1c-c3c73b287cfb-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:10:39.056862 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.056837 2543 generic.go:356] "Generic (PLEG): container finished" podID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerID="8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73" exitCode=0 Jul 24 16:10:39.057005 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.056885 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" Jul 24 16:10:39.057005 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.056914 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" event={"ID":"043b6995-4ca5-4dbe-af1c-c3c73b287cfb","Type":"ContainerDied","Data":"8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73"} Jul 24 16:10:39.057005 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.056966 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm" event={"ID":"043b6995-4ca5-4dbe-af1c-c3c73b287cfb","Type":"ContainerDied","Data":"0cbc44a5013e17d1008681339222d9e7e4d7760179a8ce5f20663eeb8592ecdd"} Jul 24 16:10:39.057005 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.056988 2543 scope.go:117] "RemoveContainer" containerID="8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73" Jul 24 16:10:39.066696 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.066679 2543 scope.go:117] "RemoveContainer" containerID="7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00" Jul 24 16:10:39.075311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.075295 2543 scope.go:117] "RemoveContainer" containerID="8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73" Jul 24 16:10:39.075597 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:10:39.075548 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73\": container with ID starting with 8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73 not found: ID does not exist" containerID="8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73" Jul 24 16:10:39.075664 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.075609 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73"} err="failed to get container status \"8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73\": rpc error: code = NotFound desc = could not find container \"8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73\": container with ID starting with 8323d07f3dd8641a7fb55ebbd043cbcfee13481306b965fc71f8b16e25ecde73 not found: ID does not exist" Jul 24 16:10:39.075664 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.075627 2543 scope.go:117] "RemoveContainer" containerID="7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00" Jul 24 16:10:39.075879 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:10:39.075854 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00\": container with ID starting with 7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00 not found: ID does not exist" containerID="7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00" Jul 24 16:10:39.075925 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.075884 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00"} err="failed to get container status \"7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00\": rpc error: code = NotFound desc = could not find container \"7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00\": container with ID starting with 7049bbb736e92970928d1810de4f336b72de9802dd0bff6acec22a5867971c00 not found: ID does not exist" Jul 24 16:10:39.076398 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.076380 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm"] Jul 24 16:10:39.080223 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:39.080199 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-54f72705-gw-openshift-default-55796f4b64-g7pgm"] Jul 24 16:10:41.045992 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:41.045949 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" path="/var/lib/kubelet/pods/043b6995-4ca5-4dbe-af1c-c3c73b287cfb/volumes" Jul 24 16:10:51.040529 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.040495 2543 patch_prober.go:28] interesting pod/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 16:10:51.040907 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.040556 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 16:10:51.126685 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.126649 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerID="3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15" exitCode=0 Jul 24 16:10:51.126813 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.126721 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" event={"ID":"cf6cd4f9-0de6-4990-b20b-865a5bb56133","Type":"ContainerDied","Data":"3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15"} Jul 24 16:10:51.127129 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.127115 2543 scope.go:117] "RemoveContainer" containerID="3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15" Jul 24 16:10:51.128230 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.128202 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:51.128311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.128296 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:51.128348 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:51.128334 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:10:52.136473 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:52.136435 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" event={"ID":"cf6cd4f9-0de6-4990-b20b-865a5bb56133","Type":"ContainerStarted","Data":"ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa"} Jul 24 16:10:52.640035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:52.640003 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:52.640272 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:52.640251 2543 patch_prober.go:28] interesting pod/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 16:10:52.640350 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:52.640330 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 16:10:53.639338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:53.639297 2543 patch_prober.go:28] interesting pod/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 16:10:53.639846 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:53.639358 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 16:10:54.639522 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:54.639487 2543 patch_prober.go:28] interesting pod/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 16:10:54.639959 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:54.639585 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 16:10:55.639794 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:55.639756 2543 patch_prober.go:28] interesting pod/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Jul 24 16:10:55.640204 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:55.639813 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Jul 24 16:10:56.643996 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:56.643966 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:56.644380 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:56.644234 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:10:56.644841 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:10:56.644825 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:11:02.267651 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.267608 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg"] Jul 24 16:11:02.268669 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.268645 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" Jul 24 16:11:02.268669 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.268670 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" Jul 24 16:11:02.268845 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.268695 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" Jul 24 16:11:02.268845 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.268703 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" Jul 24 16:11:02.268952 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.268939 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" Jul 24 16:11:02.272513 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.272487 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:02.277336 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.277303 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\"" Jul 24 16:11:02.281266 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.281230 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg"] Jul 24 16:11:02.363461 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.363411 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda55163-9dd4-4bd3-9910-fa3832d66946-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e125f3d6-55psg\" (UID: \"eda55163-9dd4-4bd3-9910-fa3832d66946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:02.464266 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.464218 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda55163-9dd4-4bd3-9910-fa3832d66946-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e125f3d6-55psg\" (UID: \"eda55163-9dd4-4bd3-9910-fa3832d66946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:02.464871 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.464853 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda55163-9dd4-4bd3-9910-fa3832d66946-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-e125f3d6-55psg\" (UID: \"eda55163-9dd4-4bd3-9910-fa3832d66946\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:02.585315 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.585230 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:02.719692 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:02.719666 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg"] Jul 24 16:11:02.721359 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:11:02.721335 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeda55163_9dd4_4bd3_9910_fa3832d66946.slice/crio-15d9b15edfbfe320cc31f3f368ee6a0c4957b8ab1946cb84d3aa5a8fd636d8e3 WatchSource:0}: Error finding container 15d9b15edfbfe320cc31f3f368ee6a0c4957b8ab1946cb84d3aa5a8fd636d8e3: Status 404 returned error can't find the container with id 15d9b15edfbfe320cc31f3f368ee6a0c4957b8ab1946cb84d3aa5a8fd636d8e3 Jul 24 16:11:03.197601 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:03.197479 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" event={"ID":"eda55163-9dd4-4bd3-9910-fa3832d66946","Type":"ContainerStarted","Data":"2911285c5a1d66d9db5ee68ca6b3a371277e6d5b3073f066bcfd833d45d319b6"} Jul 24 16:11:03.197601 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:03.197529 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" event={"ID":"eda55163-9dd4-4bd3-9910-fa3832d66946","Type":"ContainerStarted","Data":"15d9b15edfbfe320cc31f3f368ee6a0c4957b8ab1946cb84d3aa5a8fd636d8e3"} Jul 24 16:11:03.221389 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:03.221334 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" podStartSLOduration=1.221319063 podStartE2EDuration="1.221319063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:11:03.219896956 +0000 UTC m=+1794.751484561" watchObservedRunningTime="2026-07-24 16:11:03.221319063 +0000 UTC m=+1794.752906676" Jul 24 16:11:04.204167 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:04.204136 2543 generic.go:356] "Generic (PLEG): container finished" podID="eda55163-9dd4-4bd3-9910-fa3832d66946" containerID="2911285c5a1d66d9db5ee68ca6b3a371277e6d5b3073f066bcfd833d45d319b6" exitCode=0 Jul 24 16:11:04.204580 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:04.204213 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" event={"ID":"eda55163-9dd4-4bd3-9910-fa3832d66946","Type":"ContainerDied","Data":"2911285c5a1d66d9db5ee68ca6b3a371277e6d5b3073f066bcfd833d45d319b6"} Jul 24 16:11:05.355500 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:05.355479 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:05.494646 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:05.494611 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda55163-9dd4-4bd3-9910-fa3832d66946-maas-api-service-ca\") pod \"eda55163-9dd4-4bd3-9910-fa3832d66946\" (UID: \"eda55163-9dd4-4bd3-9910-fa3832d66946\") " Jul 24 16:11:05.494973 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:05.494947 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eda55163-9dd4-4bd3-9910-fa3832d66946-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eda55163-9dd4-4bd3-9910-fa3832d66946" (UID: "eda55163-9dd4-4bd3-9910-fa3832d66946"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:05.595849 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:05.595819 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eda55163-9dd4-4bd3-9910-fa3832d66946-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:06.216179 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:06.216154 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" Jul 24 16:11:06.216179 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:06.216175 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg" event={"ID":"eda55163-9dd4-4bd3-9910-fa3832d66946","Type":"ContainerDied","Data":"15d9b15edfbfe320cc31f3f368ee6a0c4957b8ab1946cb84d3aa5a8fd636d8e3"} Jul 24 16:11:06.216399 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:06.216216 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15d9b15edfbfe320cc31f3f368ee6a0c4957b8ab1946cb84d3aa5a8fd636d8e3" Jul 24 16:11:17.343983 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:17.343944 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg"] Jul 24 16:11:17.346878 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:17.346848 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e125f3d6-55psg"] Jul 24 16:11:18.546174 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:18.546131 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz"] Jul 24 16:11:18.546672 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:18.546425 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" containerID="cri-o://ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa" gracePeriod=30 Jul 24 16:11:19.043915 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.043869 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eda55163-9dd4-4bd3-9910-fa3832d66946" path="/var/lib/kubelet/pods/eda55163-9dd4-4bd3-9910-fa3832d66946/volumes" Jul 24 16:11:19.350341 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.350260 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq"] Jul 24 16:11:19.350833 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.350814 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eda55163-9dd4-4bd3-9910-fa3832d66946" containerName="revoke-keys" Jul 24 16:11:19.350833 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.350833 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="eda55163-9dd4-4bd3-9910-fa3832d66946" containerName="revoke-keys" Jul 24 16:11:19.350993 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.350939 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="043b6995-4ca5-4dbe-af1c-c3c73b287cfb" containerName="istio-proxy" Jul 24 16:11:19.350993 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.350956 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="eda55163-9dd4-4bd3-9910-fa3832d66946" containerName="revoke-keys" Jul 24 16:11:19.355284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.355264 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.357920 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.357895 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-da03131f-openshift-default-dockercfg-pj89j\"" Jul 24 16:11:19.368246 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.368226 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq"] Jul 24 16:11:19.531139 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531106 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-certs\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531158 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hldm5\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-kube-api-access-hldm5\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531186 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-envoy\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531233 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-socket\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531256 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-token\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531285 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-data\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531306 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aecc558-0c30-493a-b8ad-29a20be8910b-istiod-ca-cert\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531632 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531378 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-credential-socket\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.531632 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.531447 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-podinfo\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.632964 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.632877 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-certs\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.632964 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.632930 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hldm5\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-kube-api-access-hldm5\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.632964 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.632955 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-envoy\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.632980 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-socket\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633003 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-token\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633036 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-data\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633064 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aecc558-0c30-493a-b8ad-29a20be8910b-istiod-ca-cert\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633111 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-credential-socket\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633139 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-podinfo\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633271 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-certs\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633347 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-socket\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633969 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633639 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-credential-socket\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.633969 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.633685 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-data\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.634080 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.634014 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aecc558-0c30-493a-b8ad-29a20be8910b-istiod-ca-cert\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.635367 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.635340 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-envoy\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.635781 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.635767 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-podinfo\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.642998 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.642971 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-token\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.643111 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.643051 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hldm5\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-kube-api-access-hldm5\") pod \"e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.667109 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.667087 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:19.821171 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.821146 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq"] Jul 24 16:11:19.823070 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:11:19.823040 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aecc558_0c30_493a_b8ad_29a20be8910b.slice/crio-b182d7b54b75bb10ddc2e31554621876fd48ed115ac3e40fd9431c376aca6774 WatchSource:0}: Error finding container b182d7b54b75bb10ddc2e31554621876fd48ed115ac3e40fd9431c376aca6774: Status 404 returned error can't find the container with id b182d7b54b75bb10ddc2e31554621876fd48ed115ac3e40fd9431c376aca6774 Jul 24 16:11:19.825534 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.825499 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:11:19.825638 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.825592 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:11:19.825638 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:19.825627 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:11:20.296701 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:20.296659 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" event={"ID":"8aecc558-0c30-493a-b8ad-29a20be8910b","Type":"ContainerStarted","Data":"53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b"} Jul 24 16:11:20.296876 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:20.296723 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" event={"ID":"8aecc558-0c30-493a-b8ad-29a20be8910b","Type":"ContainerStarted","Data":"b182d7b54b75bb10ddc2e31554621876fd48ed115ac3e40fd9431c376aca6774"} Jul 24 16:11:20.323314 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:20.323260 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" podStartSLOduration=1.323243382 podStartE2EDuration="1.323243382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:11:20.321516201 +0000 UTC m=+1811.853103814" watchObservedRunningTime="2026-07-24 16:11:20.323243382 +0000 UTC m=+1811.854831044" Jul 24 16:11:20.668341 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:20.668268 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:20.673165 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:20.673142 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:21.302098 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:21.302065 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:21.302899 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:21.302878 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:23.813097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.813037 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:11:23.872680 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872653 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-podinfo\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.872856 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872689 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istiod-ca-cert\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.872856 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872718 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-certs\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.872856 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872742 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-token\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.872856 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872791 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-credential-socket\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.872856 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872812 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-socket\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.872856 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872836 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrz87\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-kube-api-access-lrz87\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.873158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872865 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-envoy\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.873158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.872934 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-data\") pod \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\" (UID: \"cf6cd4f9-0de6-4990-b20b-865a5bb56133\") " Jul 24 16:11:23.873158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873003 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:11:23.873158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873046 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:23.873158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873105 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:11:23.873392 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873331 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.873392 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873350 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.873392 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873365 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.873392 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.873346 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:11:23.874028 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.874004 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-data" (OuterVolumeSpecName: "istio-data") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:11:23.875210 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.875191 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:11:23.875306 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.875232 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-kube-api-access-lrz87" (OuterVolumeSpecName: "kube-api-access-lrz87") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "kube-api-access-lrz87". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:11:23.875491 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.875469 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:11:23.875617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.875496 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-token" (OuterVolumeSpecName: "istio-token") pod "cf6cd4f9-0de6-4990-b20b-865a5bb56133" (UID: "cf6cd4f9-0de6-4990-b20b-865a5bb56133"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:11:23.974136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.974106 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.974136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.974136 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.974301 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.974150 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.974301 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.974170 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrz87\" (UniqueName: \"kubernetes.io/projected/cf6cd4f9-0de6-4990-b20b-865a5bb56133-kube-api-access-lrz87\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.974301 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.974184 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:23.974301 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:23.974197 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6cd4f9-0de6-4990-b20b-865a5bb56133-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:24.323739 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.323703 2543 generic.go:356] "Generic (PLEG): container finished" podID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerID="ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa" exitCode=0 Jul 24 16:11:24.323928 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.323760 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" Jul 24 16:11:24.323928 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.323788 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" event={"ID":"cf6cd4f9-0de6-4990-b20b-865a5bb56133","Type":"ContainerDied","Data":"ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa"} Jul 24 16:11:24.323928 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.323844 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz" event={"ID":"cf6cd4f9-0de6-4990-b20b-865a5bb56133","Type":"ContainerDied","Data":"b9fb432f49c745e912c9ba07050aab3ffd934fed5b64ae2878a0e9b1ff02e0d6"} Jul 24 16:11:24.323928 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.323876 2543 scope.go:117] "RemoveContainer" containerID="ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa" Jul 24 16:11:24.333833 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.333813 2543 scope.go:117] "RemoveContainer" containerID="3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15" Jul 24 16:11:24.345589 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.345462 2543 scope.go:117] "RemoveContainer" containerID="ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa" Jul 24 16:11:24.345816 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:11:24.345792 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa\": container with ID starting with ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa not found: ID does not exist" containerID="ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa" Jul 24 16:11:24.345867 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.345847 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa"} err="failed to get container status \"ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa\": rpc error: code = NotFound desc = could not find container \"ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa\": container with ID starting with ccd20d88967458c324aa3716590c650852ceff610c9426217c5baf386e12c6fa not found: ID does not exist" Jul 24 16:11:24.345910 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.345870 2543 scope.go:117] "RemoveContainer" containerID="3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15" Jul 24 16:11:24.346157 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:11:24.346136 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15\": container with ID starting with 3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15 not found: ID does not exist" containerID="3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15" Jul 24 16:11:24.346224 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.346164 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15"} err="failed to get container status \"3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15\": rpc error: code = NotFound desc = could not find container \"3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15\": container with ID starting with 3bf859909fbb46d5a3bca5e7e2923a4d92077f4a1f2e8b18287b4d87fe857a15 not found: ID does not exist" Jul 24 16:11:24.347449 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.347427 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz"] Jul 24 16:11:24.352429 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:24.352407 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e125f3d6-openshift-default-77bbd49fd4-rbzgz"] Jul 24 16:11:25.043672 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:25.043643 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" path="/var/lib/kubelet/pods/cf6cd4f9-0de6-4990-b20b-865a5bb56133/volumes" Jul 24 16:11:35.387791 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:35.387705 2543 generic.go:356] "Generic (PLEG): container finished" podID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerID="53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b" exitCode=0 Jul 24 16:11:35.387791 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:35.387744 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" event={"ID":"8aecc558-0c30-493a-b8ad-29a20be8910b","Type":"ContainerDied","Data":"53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b"} Jul 24 16:11:35.388220 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:35.388132 2543 scope.go:117] "RemoveContainer" containerID="53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b" Jul 24 16:11:35.390933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:35.389637 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:11:35.390933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:35.389900 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:11:35.390933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:35.390017 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:11:36.302026 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:36.301982 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:36.395471 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:36.395434 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" event={"ID":"8aecc558-0c30-493a-b8ad-29a20be8910b","Type":"ContainerStarted","Data":"ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5"} Jul 24 16:11:36.667554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:36.667463 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:36.668738 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:36.668710 2543 patch_prober.go:28] interesting pod/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 16:11:36.668863 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:36.668774 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 16:11:37.667579 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:37.667534 2543 patch_prober.go:28] interesting pod/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 16:11:37.667983 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:37.667621 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 16:11:38.667429 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:38.667396 2543 patch_prober.go:28] interesting pod/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 16:11:38.667626 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:38.667449 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 16:11:39.668090 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:39.668051 2543 patch_prober.go:28] interesting pod/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 24 16:11:39.668484 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:39.668125 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 24 16:11:40.671331 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:40.671285 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:40.671801 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:40.671590 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:40.672232 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:40.672212 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:11:46.592980 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.592945 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877"] Jul 24 16:11:46.593456 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.593442 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" Jul 24 16:11:46.593508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.593458 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" Jul 24 16:11:46.593508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.593475 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" Jul 24 16:11:46.593508 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.593481 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" Jul 24 16:11:46.593617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.593591 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" Jul 24 16:11:46.593617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.593598 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf6cd4f9-0de6-4990-b20b-865a5bb56133" containerName="istio-proxy" Jul 24 16:11:46.595526 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.595509 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:46.597988 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.597965 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\"" Jul 24 16:11:46.603209 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.603185 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877"] Jul 24 16:11:46.711891 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.711857 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0216fd49-a3e3-499a-8d38-fad1dd665f4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-da03131f-jj877\" (UID: \"0216fd49-a3e3-499a-8d38-fad1dd665f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:46.812623 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.812589 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0216fd49-a3e3-499a-8d38-fad1dd665f4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-da03131f-jj877\" (UID: \"0216fd49-a3e3-499a-8d38-fad1dd665f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:46.813141 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.813120 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0216fd49-a3e3-499a-8d38-fad1dd665f4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-da03131f-jj877\" (UID: \"0216fd49-a3e3-499a-8d38-fad1dd665f4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:46.907489 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:46.907411 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:47.053201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:47.053059 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877"] Jul 24 16:11:47.055408 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:11:47.055372 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0216fd49_a3e3_499a_8d38_fad1dd665f4d.slice/crio-249935b27091749289acc96e79ca32744b9928f7119e97fc820ca03c1b0be3b9 WatchSource:0}: Error finding container 249935b27091749289acc96e79ca32744b9928f7119e97fc820ca03c1b0be3b9: Status 404 returned error can't find the container with id 249935b27091749289acc96e79ca32744b9928f7119e97fc820ca03c1b0be3b9 Jul 24 16:11:47.458297 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:47.458249 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" event={"ID":"0216fd49-a3e3-499a-8d38-fad1dd665f4d","Type":"ContainerStarted","Data":"40ebaed8e9f739103d4cf0b32d767d9a2d89845a58a52a9ba0913f66b82f3e7a"} Jul 24 16:11:47.458483 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:47.458359 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" event={"ID":"0216fd49-a3e3-499a-8d38-fad1dd665f4d","Type":"ContainerStarted","Data":"249935b27091749289acc96e79ca32744b9928f7119e97fc820ca03c1b0be3b9"} Jul 24 16:11:47.474252 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:47.474197 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" podStartSLOduration=1.474181223 podStartE2EDuration="1.474181223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:11:47.47230988 +0000 UTC m=+1839.003897518" watchObservedRunningTime="2026-07-24 16:11:47.474181223 +0000 UTC m=+1839.005768856" Jul 24 16:11:48.465603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:48.465548 2543 generic.go:356] "Generic (PLEG): container finished" podID="0216fd49-a3e3-499a-8d38-fad1dd665f4d" containerID="40ebaed8e9f739103d4cf0b32d767d9a2d89845a58a52a9ba0913f66b82f3e7a" exitCode=0 Jul 24 16:11:48.465603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:48.465589 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" event={"ID":"0216fd49-a3e3-499a-8d38-fad1dd665f4d","Type":"ContainerDied","Data":"40ebaed8e9f739103d4cf0b32d767d9a2d89845a58a52a9ba0913f66b82f3e7a"} Jul 24 16:11:49.607602 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:49.607577 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:49.744201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:49.744127 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0216fd49-a3e3-499a-8d38-fad1dd665f4d-maas-api-service-ca\") pod \"0216fd49-a3e3-499a-8d38-fad1dd665f4d\" (UID: \"0216fd49-a3e3-499a-8d38-fad1dd665f4d\") " Jul 24 16:11:49.744460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:49.744439 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0216fd49-a3e3-499a-8d38-fad1dd665f4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0216fd49-a3e3-499a-8d38-fad1dd665f4d" (UID: "0216fd49-a3e3-499a-8d38-fad1dd665f4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:11:49.844999 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:49.844972 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0216fd49-a3e3-499a-8d38-fad1dd665f4d-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:11:50.478075 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:50.478046 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" Jul 24 16:11:50.478281 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:50.478075 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877" event={"ID":"0216fd49-a3e3-499a-8d38-fad1dd665f4d","Type":"ContainerDied","Data":"249935b27091749289acc96e79ca32744b9928f7119e97fc820ca03c1b0be3b9"} Jul 24 16:11:50.478281 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:11:50.478120 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="249935b27091749289acc96e79ca32744b9928f7119e97fc820ca03c1b0be3b9" Jul 24 16:12:01.695650 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:01.695616 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877"] Jul 24 16:12:01.696153 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:01.696025 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-da03131f-jj877"] Jul 24 16:12:01.826687 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:01.826651 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq"] Jul 24 16:12:01.826975 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:01.826939 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" containerID="cri-o://ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5" gracePeriod=30 Jul 24 16:12:03.043622 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:03.043588 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0216fd49-a3e3-499a-8d38-fad1dd665f4d" path="/var/lib/kubelet/pods/0216fd49-a3e3-499a-8d38-fad1dd665f4d/volumes" Jul 24 16:12:07.094247 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.094190 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:12:07.112136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112109 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-podinfo\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112275 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112153 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-token\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112275 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112189 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-socket\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112275 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112214 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aecc558-0c30-493a-b8ad-29a20be8910b-istiod-ca-cert\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112442 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112284 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-envoy\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112442 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112332 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-credential-socket\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112442 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112362 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-certs\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112442 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112392 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-data\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112442 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112418 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hldm5\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-kube-api-access-hldm5\") pod \"8aecc558-0c30-493a-b8ad-29a20be8910b\" (UID: \"8aecc558-0c30-493a-b8ad-29a20be8910b\") " Jul 24 16:12:07.112729 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112535 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:07.112789 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112726 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:07.112991 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112967 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:07.113077 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.112998 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.113077 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.113017 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.113182 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.113132 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aecc558-0c30-493a-b8ad-29a20be8910b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:12:07.113406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.113375 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-data" (OuterVolumeSpecName: "istio-data") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:07.115522 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.115150 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-kube-api-access-hldm5" (OuterVolumeSpecName: "kube-api-access-hldm5") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "kube-api-access-hldm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:12:07.115522 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.115226 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:12:07.115522 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.115318 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:12:07.115522 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.115390 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-token" (OuterVolumeSpecName: "istio-token") pod "8aecc558-0c30-493a-b8ad-29a20be8910b" (UID: "8aecc558-0c30-493a-b8ad-29a20be8910b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:12:07.214063 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214029 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.214063 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214056 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.214063 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214065 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hldm5\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-kube-api-access-hldm5\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.214284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214075 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.214284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214083 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.214284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214091 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8aecc558-0c30-493a-b8ad-29a20be8910b-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.214284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.214098 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8aecc558-0c30-493a-b8ad-29a20be8910b-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:12:07.573846 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.573814 2543 generic.go:356] "Generic (PLEG): container finished" podID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerID="ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5" exitCode=0 Jul 24 16:12:07.574014 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.573864 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" Jul 24 16:12:07.574014 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.573868 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" event={"ID":"8aecc558-0c30-493a-b8ad-29a20be8910b","Type":"ContainerDied","Data":"ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5"} Jul 24 16:12:07.574014 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.573912 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq" event={"ID":"8aecc558-0c30-493a-b8ad-29a20be8910b","Type":"ContainerDied","Data":"b182d7b54b75bb10ddc2e31554621876fd48ed115ac3e40fd9431c376aca6774"} Jul 24 16:12:07.574014 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.573933 2543 scope.go:117] "RemoveContainer" containerID="ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5" Jul 24 16:12:07.583455 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.583437 2543 scope.go:117] "RemoveContainer" containerID="53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b" Jul 24 16:12:07.591628 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.591612 2543 scope.go:117] "RemoveContainer" containerID="ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5" Jul 24 16:12:07.591916 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:12:07.591891 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5\": container with ID starting with ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5 not found: ID does not exist" containerID="ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5" Jul 24 16:12:07.592006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.591929 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5"} err="failed to get container status \"ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5\": rpc error: code = NotFound desc = could not find container \"ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5\": container with ID starting with ac6d9594a295ce2014f57172a20db94d81d2e09e11ef10e83825a77dea7cf5f5 not found: ID does not exist" Jul 24 16:12:07.592006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.591955 2543 scope.go:117] "RemoveContainer" containerID="53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b" Jul 24 16:12:07.592297 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:12:07.592273 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b\": container with ID starting with 53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b not found: ID does not exist" containerID="53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b" Jul 24 16:12:07.592410 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.592305 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b"} err="failed to get container status \"53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b\": rpc error: code = NotFound desc = could not find container \"53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b\": container with ID starting with 53d387e33530d66e3afdc1fe8125366df398f82c684dee375f925781ea85867b not found: ID does not exist" Jul 24 16:12:07.595117 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.595095 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq"] Jul 24 16:12:07.600353 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:07.600333 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-da03131f-openshift-default-67f87bbd5b-kgdlq"] Jul 24 16:12:09.044359 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:12:09.044315 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" path="/var/lib/kubelet/pods/8aecc558-0c30-493a-b8ad-29a20be8910b/volumes" Jul 24 16:13:39.555410 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.555380 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh"] Jul 24 16:13:39.555991 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.555962 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" Jul 24 16:13:39.555991 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.555983 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" Jul 24 16:13:39.555991 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.555993 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" Jul 24 16:13:39.555991 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.556002 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" Jul 24 16:13:39.556244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.556018 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0216fd49-a3e3-499a-8d38-fad1dd665f4d" containerName="revoke-keys" Jul 24 16:13:39.556244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.556026 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0216fd49-a3e3-499a-8d38-fad1dd665f4d" containerName="revoke-keys" Jul 24 16:13:39.556244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.556106 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" Jul 24 16:13:39.556244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.556118 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0216fd49-a3e3-499a-8d38-fad1dd665f4d" containerName="revoke-keys" Jul 24 16:13:39.559417 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.559398 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.561765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.561748 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4604a53f-openshift-default-dockercfg-qdc56\"" Jul 24 16:13:39.579658 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.579637 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh"] Jul 24 16:13:39.711737 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.711698 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxskj\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-kube-api-access-pxskj\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.711932 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.711752 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-certs\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.711932 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.711865 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-socket\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.711932 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.711914 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-data\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.712117 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.712004 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e016a7-4290-490e-bf13-3e43c264f95f-istiod-ca-cert\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.712117 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.712046 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-istio-token\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.712228 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.712116 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-credential-socket\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.712228 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.712135 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e016a7-4290-490e-bf13-3e43c264f95f-istio-podinfo\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.712228 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.712154 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-envoy\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813574 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813468 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-certs\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813574 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813531 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-socket\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813581 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-data\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813623 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e016a7-4290-490e-bf13-3e43c264f95f-istiod-ca-cert\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813653 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-istio-token\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813691 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-credential-socket\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813717 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e016a7-4290-490e-bf13-3e43c264f95f-istio-podinfo\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.813795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813744 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-envoy\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.814082 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813839 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxskj\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-kube-api-access-pxskj\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.814082 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.813937 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-certs\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.814193 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.814128 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-credential-socket\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.814404 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.814374 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e016a7-4290-490e-bf13-3e43c264f95f-istiod-ca-cert\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.814404 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.814397 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-socket\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.814654 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.814499 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-data\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.816381 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.816354 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-envoy\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.816589 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.816548 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e016a7-4290-490e-bf13-3e43c264f95f-istio-podinfo\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.822257 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.822233 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxskj\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-kube-api-access-pxskj\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.822355 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.822287 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-istio-token\") pod \"e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:39.869888 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:39.869854 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:40.018725 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:40.018694 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh"] Jul 24 16:13:40.027213 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:13:40.022373 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e016a7_4290_490e_bf13_3e43c264f95f.slice/crio-8040a55c5bda1fb10dfbedf3b12158ebde2a111e6ce851758eb3dd31ab70c9bf WatchSource:0}: Error finding container 8040a55c5bda1fb10dfbedf3b12158ebde2a111e6ce851758eb3dd31ab70c9bf: Status 404 returned error can't find the container with id 8040a55c5bda1fb10dfbedf3b12158ebde2a111e6ce851758eb3dd31ab70c9bf Jul 24 16:13:40.030160 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:40.030126 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:13:40.030316 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:40.030209 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:13:40.030316 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:40.030257 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:13:40.072181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:40.072153 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" event={"ID":"94e016a7-4290-490e-bf13-3e43c264f95f","Type":"ContainerStarted","Data":"8040a55c5bda1fb10dfbedf3b12158ebde2a111e6ce851758eb3dd31ab70c9bf"} Jul 24 16:13:41.079252 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:41.079213 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" event={"ID":"94e016a7-4290-490e-bf13-3e43c264f95f","Type":"ContainerStarted","Data":"f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e"} Jul 24 16:13:41.116364 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:41.116307 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podStartSLOduration=2.116290139 podStartE2EDuration="2.116290139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:13:41.11452047 +0000 UTC m=+1952.646108084" watchObservedRunningTime="2026-07-24 16:13:41.116290139 +0000 UTC m=+1952.647877751" Jul 24 16:13:41.870603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:41.870553 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:41.875388 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:41.875363 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:42.085133 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:42.085104 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:42.086154 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:42.086131 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:54.888655 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:54.888617 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr"] Jul 24 16:13:54.889299 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:54.889282 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aecc558-0c30-493a-b8ad-29a20be8910b" containerName="istio-proxy" Jul 24 16:13:54.892442 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:54.892427 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:54.895060 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:54.895039 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-b924x\"" Jul 24 16:13:54.899384 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:54.899360 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr"] Jul 24 16:13:54.972829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:54.972796 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76d841a-3086-4414-a6ba-7ba76382dd7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr\" (UID: \"f76d841a-3086-4414-a6ba-7ba76382dd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:55.073521 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:55.073490 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76d841a-3086-4414-a6ba-7ba76382dd7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr\" (UID: \"f76d841a-3086-4414-a6ba-7ba76382dd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:55.074154 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:55.074133 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76d841a-3086-4414-a6ba-7ba76382dd7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr\" (UID: \"f76d841a-3086-4414-a6ba-7ba76382dd7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:55.208036 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:55.207944 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:55.351362 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:55.351316 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr"] Jul 24 16:13:55.352098 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:13:55.352072 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76d841a_3086_4414_a6ba_7ba76382dd7e.slice/crio-112648a78ce57cb3c11cc2cb4c99aa86d3e2564c7cd28ab94ae12f1d634a38e1 WatchSource:0}: Error finding container 112648a78ce57cb3c11cc2cb4c99aa86d3e2564c7cd28ab94ae12f1d634a38e1: Status 404 returned error can't find the container with id 112648a78ce57cb3c11cc2cb4c99aa86d3e2564c7cd28ab94ae12f1d634a38e1 Jul 24 16:13:56.170014 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.169926 2543 generic.go:356] "Generic (PLEG): container finished" podID="94e016a7-4290-490e-bf13-3e43c264f95f" containerID="f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e" exitCode=0 Jul 24 16:13:56.170014 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.169948 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" event={"ID":"94e016a7-4290-490e-bf13-3e43c264f95f","Type":"ContainerDied","Data":"f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e"} Jul 24 16:13:56.170760 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.170498 2543 scope.go:117] "RemoveContainer" containerID="f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e" Jul 24 16:13:56.171674 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.171644 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:13:56.171809 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.171729 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:13:56.171809 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.171773 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:13:56.172181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.172106 2543 generic.go:356] "Generic (PLEG): container finished" podID="f76d841a-3086-4414-a6ba-7ba76382dd7e" containerID="2032b7c40645da96abd0548b3957d77dab3b233f6da67d270580b7d537a6b7fe" exitCode=0 Jul 24 16:13:56.172181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.172150 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" event={"ID":"f76d841a-3086-4414-a6ba-7ba76382dd7e","Type":"ContainerDied","Data":"2032b7c40645da96abd0548b3957d77dab3b233f6da67d270580b7d537a6b7fe"} Jul 24 16:13:56.172327 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:56.172202 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" event={"ID":"f76d841a-3086-4414-a6ba-7ba76382dd7e","Type":"ContainerStarted","Data":"112648a78ce57cb3c11cc2cb4c99aa86d3e2564c7cd28ab94ae12f1d634a38e1"} Jul 24 16:13:57.086025 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.085996 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:57.183105 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.183069 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" event={"ID":"94e016a7-4290-490e-bf13-3e43c264f95f","Type":"ContainerStarted","Data":"8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094"} Jul 24 16:13:57.357295 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.357274 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:57.400203 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.400177 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76d841a-3086-4414-a6ba-7ba76382dd7e-maas-api-service-ca\") pod \"f76d841a-3086-4414-a6ba-7ba76382dd7e\" (UID: \"f76d841a-3086-4414-a6ba-7ba76382dd7e\") " Jul 24 16:13:57.400510 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.400488 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76d841a-3086-4414-a6ba-7ba76382dd7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f76d841a-3086-4414-a6ba-7ba76382dd7e" (UID: "f76d841a-3086-4414-a6ba-7ba76382dd7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:13:57.501998 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.501959 2543 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f76d841a-3086-4414-a6ba-7ba76382dd7e-maas-api-service-ca\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:13:57.870671 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.870638 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:13:57.871937 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.871911 2543 patch_prober.go:28] interesting pod/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 16:13:57.872072 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:57.871959 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 16:13:58.190371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:58.190284 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" Jul 24 16:13:58.190371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:58.190309 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr" event={"ID":"f76d841a-3086-4414-a6ba-7ba76382dd7e","Type":"ContainerDied","Data":"112648a78ce57cb3c11cc2cb4c99aa86d3e2564c7cd28ab94ae12f1d634a38e1"} Jul 24 16:13:58.190371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:58.190364 2543 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="112648a78ce57cb3c11cc2cb4c99aa86d3e2564c7cd28ab94ae12f1d634a38e1" Jul 24 16:13:58.870971 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:58.870937 2543 patch_prober.go:28] interesting pod/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 16:13:58.871155 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:58.870993 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 16:13:59.870694 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:59.870657 2543 patch_prober.go:28] interesting pod/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 16:13:59.871180 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:13:59.870727 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 16:14:00.870383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:00.870346 2543 patch_prober.go:28] interesting pod/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 24 16:14:00.870607 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:00.870403 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 24 16:14:01.874169 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:01.874138 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:14:01.874582 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:01.874420 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:14:01.875200 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:01.875178 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:14:09.968958 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:09.968920 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr"] Jul 24 16:14:09.972019 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:09.971994 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4604a53f-cp4tr"] Jul 24 16:14:11.044324 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:11.044292 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f76d841a-3086-4414-a6ba-7ba76382dd7e" path="/var/lib/kubelet/pods/f76d841a-3086-4414-a6ba-7ba76382dd7e/volumes" Jul 24 16:14:16.349874 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:16.349841 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh"] Jul 24 16:14:16.350327 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:16.350045 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" containerID="cri-o://8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094" gracePeriod=30 Jul 24 16:14:16.876194 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:16.876156 2543 patch_prober.go:28] interesting pod/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 16:14:16.876368 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:16.876233 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 16:14:21.613243 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.613185 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:14:21.757517 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757480 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-certs\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.757742 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757538 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-credential-socket\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.757742 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757619 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-data\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.757742 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757645 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-socket\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.757742 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757681 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-envoy\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757759 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e016a7-4290-490e-bf13-3e43c264f95f-istiod-ca-cert\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757808 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-istio-token\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757829 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757841 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e016a7-4290-490e-bf13-3e43c264f95f-istio-podinfo\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757893 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.757940 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxskj\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-kube-api-access-pxskj\") pod \"94e016a7-4290-490e-bf13-3e43c264f95f\" (UID: \"94e016a7-4290-490e-bf13-3e43c264f95f\") " Jul 24 16:14:21.758038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758012 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:14:21.758377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758231 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94e016a7-4290-490e-bf13-3e43c264f95f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:14:21.758377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758353 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.758377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758372 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.758520 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758385 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.758520 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758398 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94e016a7-4290-490e-bf13-3e43c264f95f-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.758836 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.758813 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-data" (OuterVolumeSpecName: "istio-data") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:14:21.760184 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.760160 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:14:21.760345 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.760322 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-kube-api-access-pxskj" (OuterVolumeSpecName: "kube-api-access-pxskj") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "kube-api-access-pxskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:14:21.760650 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.760626 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94e016a7-4290-490e-bf13-3e43c264f95f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:14:21.760650 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.760633 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-istio-token" (OuterVolumeSpecName: "istio-token") pod "94e016a7-4290-490e-bf13-3e43c264f95f" (UID: "94e016a7-4290-490e-bf13-3e43c264f95f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:14:21.859435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.859403 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94e016a7-4290-490e-bf13-3e43c264f95f-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.859435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.859431 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxskj\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-kube-api-access-pxskj\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.859435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.859441 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.859705 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.859450 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94e016a7-4290-490e-bf13-3e43c264f95f-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:21.859705 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:21.859457 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94e016a7-4290-490e-bf13-3e43c264f95f-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:14:22.338278 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.338245 2543 generic.go:356] "Generic (PLEG): container finished" podID="94e016a7-4290-490e-bf13-3e43c264f95f" containerID="8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094" exitCode=0 Jul 24 16:14:22.338463 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.338301 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" Jul 24 16:14:22.338463 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.338340 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" event={"ID":"94e016a7-4290-490e-bf13-3e43c264f95f","Type":"ContainerDied","Data":"8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094"} Jul 24 16:14:22.338463 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.338393 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh" event={"ID":"94e016a7-4290-490e-bf13-3e43c264f95f","Type":"ContainerDied","Data":"8040a55c5bda1fb10dfbedf3b12158ebde2a111e6ce851758eb3dd31ab70c9bf"} Jul 24 16:14:22.338463 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.338415 2543 scope.go:117] "RemoveContainer" containerID="8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094" Jul 24 16:14:22.348472 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.348451 2543 scope.go:117] "RemoveContainer" containerID="f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e" Jul 24 16:14:22.356762 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.356741 2543 scope.go:117] "RemoveContainer" containerID="8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094" Jul 24 16:14:22.357125 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:14:22.357095 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094\": container with ID starting with 8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094 not found: ID does not exist" containerID="8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094" Jul 24 16:14:22.357255 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.357138 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094"} err="failed to get container status \"8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094\": rpc error: code = NotFound desc = could not find container \"8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094\": container with ID starting with 8b79d139ab8a59c1139f7ae1eb75abf91a62f269125e3f47bbf16ab08c4b6094 not found: ID does not exist" Jul 24 16:14:22.357255 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.357162 2543 scope.go:117] "RemoveContainer" containerID="f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e" Jul 24 16:14:22.357849 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:14:22.357826 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e\": container with ID starting with f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e not found: ID does not exist" containerID="f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e" Jul 24 16:14:22.357994 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.357965 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e"} err="failed to get container status \"f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e\": rpc error: code = NotFound desc = could not find container \"f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e\": container with ID starting with f063a9121ea8835e3c7d06da9074dbfc59d74457474d48d7569d5b057d75c76e not found: ID does not exist" Jul 24 16:14:22.360147 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.360126 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh"] Jul 24 16:14:22.364016 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:22.363996 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4604a53f-openshift-default-7c65b768cb-s2tvh"] Jul 24 16:14:23.045144 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:23.045115 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" path="/var/lib/kubelet/pods/94e016a7-4290-490e-bf13-3e43c264f95f/volumes" Jul 24 16:14:36.009976 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.009940 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz"] Jul 24 16:14:36.010662 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010634 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" Jul 24 16:14:36.010662 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010658 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010671 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f76d841a-3086-4414-a6ba-7ba76382dd7e" containerName="revoke-keys" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010678 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="f76d841a-3086-4414-a6ba-7ba76382dd7e" containerName="revoke-keys" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010684 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010689 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010781 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010790 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="94e016a7-4290-490e-bf13-3e43c264f95f" containerName="istio-proxy" Jul 24 16:14:36.010868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.010800 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="f76d841a-3086-4414-a6ba-7ba76382dd7e" containerName="revoke-keys" Jul 24 16:14:36.014284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.014267 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.016799 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.016779 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b89f991b-openshift-default-dockercfg-grnsr\"" Jul 24 16:14:36.025276 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.025256 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz"] Jul 24 16:14:36.096422 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.096392 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-istio-token\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.096686 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.096449 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qgj2\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-kube-api-access-2qgj2\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.096686 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.096605 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-data\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.096686 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.096683 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db348170-ad5a-4c89-8a2f-0643c03955da-istiod-ca-cert\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.096919 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.096713 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-certs\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.096919 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.096739 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-credential-socket\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.097178 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.097147 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-socket\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.097279 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.097213 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-envoy\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.097279 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.097260 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db348170-ad5a-4c89-8a2f-0643c03955da-istio-podinfo\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198068 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198034 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-socket\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198081 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-envoy\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198098 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db348170-ad5a-4c89-8a2f-0643c03955da-istio-podinfo\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198137 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-istio-token\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198162 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qgj2\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-kube-api-access-2qgj2\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198200 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-data\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198224 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db348170-ad5a-4c89-8a2f-0643c03955da-istiod-ca-cert\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198261 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198251 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-certs\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198672 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198281 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-credential-socket\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198672 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198632 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-data\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198788 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198727 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-credential-socket\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198788 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198756 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-socket\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.198974 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.198952 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-certs\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.199285 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.199261 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db348170-ad5a-4c89-8a2f-0643c03955da-istiod-ca-cert\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.200384 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.200360 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-envoy\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.200729 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.200713 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db348170-ad5a-4c89-8a2f-0643c03955da-istio-podinfo\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.206306 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.206267 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-istio-token\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.206865 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.206843 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qgj2\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-kube-api-access-2qgj2\") pod \"e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.325655 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.325538 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:36.477056 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.475219 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz"] Jul 24 16:14:36.479482 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.479426 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:14:36.479640 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.479611 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:14:36.482881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:36.479705 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:14:37.422198 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:37.422155 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" event={"ID":"db348170-ad5a-4c89-8a2f-0643c03955da","Type":"ContainerStarted","Data":"aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a"} Jul 24 16:14:37.422588 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:37.422218 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" event={"ID":"db348170-ad5a-4c89-8a2f-0643c03955da","Type":"ContainerStarted","Data":"39995721987d82e204bf0d315b8c5cd1bc4a28648c57b3fd85be71afb01c83f2"} Jul 24 16:14:37.442998 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:37.442944 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podStartSLOduration=2.442929935 podStartE2EDuration="2.442929935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:14:37.44178617 +0000 UTC m=+2008.973373805" watchObservedRunningTime="2026-07-24 16:14:37.442929935 +0000 UTC m=+2008.974517549" Jul 24 16:14:38.326747 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:38.326711 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:38.331611 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:38.331587 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:38.427617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:38.427587 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:38.428473 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:38.428455 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:40.185728 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.185698 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q"] Jul 24 16:14:40.188661 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.188640 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.194186 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.194169 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-b89f991b\"" Jul 24 16:14:40.194324 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.194308 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-b89f991b-dockercfg-7pz54\"" Jul 24 16:14:40.202858 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.202835 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q"] Jul 24 16:14:40.240222 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.240194 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-plugins-config-volume\") pod \"payload-processing-e2e-mt-b89f991b-85fc458466-nv92q\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.240388 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.240279 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbsf6\" (UniqueName: \"kubernetes.io/projected/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-kube-api-access-zbsf6\") pod \"payload-processing-e2e-mt-b89f991b-85fc458466-nv92q\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.341790 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.341749 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-plugins-config-volume\") pod \"payload-processing-e2e-mt-b89f991b-85fc458466-nv92q\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.341970 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.341812 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbsf6\" (UniqueName: \"kubernetes.io/projected/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-kube-api-access-zbsf6\") pod \"payload-processing-e2e-mt-b89f991b-85fc458466-nv92q\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.342437 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.342418 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-plugins-config-volume\") pod \"payload-processing-e2e-mt-b89f991b-85fc458466-nv92q\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.351437 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.351415 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbsf6\" (UniqueName: \"kubernetes.io/projected/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-kube-api-access-zbsf6\") pod \"payload-processing-e2e-mt-b89f991b-85fc458466-nv92q\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.499402 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.499372 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:40.843509 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:40.843477 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q"] Jul 24 16:14:40.845497 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:14:40.845465 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a0f50e_5f3a_488e_a1f9_0501fe243d0b.slice/crio-55dd4e57c2350139426d2392436c7ae757ff2581ca8d7a6d09ea9528d7fe106b WatchSource:0}: Error finding container 55dd4e57c2350139426d2392436c7ae757ff2581ca8d7a6d09ea9528d7fe106b: Status 404 returned error can't find the container with id 55dd4e57c2350139426d2392436c7ae757ff2581ca8d7a6d09ea9528d7fe106b Jul 24 16:14:41.447513 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:41.447460 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" event={"ID":"50a0f50e-5f3a-488e-a1f9-0501fe243d0b","Type":"ContainerStarted","Data":"f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d"} Jul 24 16:14:41.447953 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:41.447556 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" event={"ID":"50a0f50e-5f3a-488e-a1f9-0501fe243d0b","Type":"ContainerStarted","Data":"55dd4e57c2350139426d2392436c7ae757ff2581ca8d7a6d09ea9528d7fe106b"} Jul 24 16:14:41.447953 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:41.447604 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:41.465209 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:41.465167 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" podStartSLOduration=1.465152584 podStartE2EDuration="1.465152584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:14:41.464129773 +0000 UTC m=+2012.995717402" watchObservedRunningTime="2026-07-24 16:14:41.465152584 +0000 UTC m=+2012.996740196" Jul 24 16:14:52.454297 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:52.454268 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:14:54.195980 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.194985 2543 patch_prober.go:28] interesting pod/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:35182->10.134.0.54:15021: read: connection reset by peer" start-of-body= Jul 24 16:14:54.195980 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.195049 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": read tcp 10.134.0.2:35182->10.134.0.54:15021: read: connection reset by peer" Jul 24 16:14:54.519668 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.519635 2543 generic.go:356] "Generic (PLEG): container finished" podID="db348170-ad5a-4c89-8a2f-0643c03955da" containerID="aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a" exitCode=0 Jul 24 16:14:54.519849 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.519687 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" event={"ID":"db348170-ad5a-4c89-8a2f-0643c03955da","Type":"ContainerDied","Data":"aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a"} Jul 24 16:14:54.520119 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.520103 2543 scope.go:117] "RemoveContainer" containerID="aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a" Jul 24 16:14:54.521256 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.521226 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:14:54.521317 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.521303 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:14:54.521362 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:54.521336 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:14:55.533933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:55.533887 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" event={"ID":"db348170-ad5a-4c89-8a2f-0643c03955da","Type":"ContainerStarted","Data":"74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa"} Jul 24 16:14:56.326383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:56.326290 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:14:56.327000 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:56.326971 2543 patch_prober.go:28] interesting pod/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 16:14:56.327200 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:56.327173 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 16:14:57.327006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:57.326964 2543 patch_prober.go:28] interesting pod/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 16:14:57.327754 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:57.327719 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 16:14:58.326228 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:58.326188 2543 patch_prober.go:28] interesting pod/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 16:14:58.326528 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:58.326500 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 16:14:59.326487 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:59.326452 2543 patch_prober.go:28] interesting pod/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 24 16:14:59.327108 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:14:59.327077 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 24 16:15:00.329574 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:00.329538 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:15:00.330049 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:00.329955 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:15:00.330538 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:00.330517 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:15:44.311493 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.311404 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q"] Jul 24 16:15:44.312115 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.311676 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" podUID="50a0f50e-5f3a-488e-a1f9-0501fe243d0b" containerName="payload-processing" containerID="cri-o://f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d" gracePeriod=30 Jul 24 16:15:44.578336 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.578309 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:15:44.588048 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.588022 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-plugins-config-volume\") pod \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " Jul 24 16:15:44.588185 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.588168 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zbsf6\" (UniqueName: \"kubernetes.io/projected/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-kube-api-access-zbsf6\") pod \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\" (UID: \"50a0f50e-5f3a-488e-a1f9-0501fe243d0b\") " Jul 24 16:15:44.588331 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.588309 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "50a0f50e-5f3a-488e-a1f9-0501fe243d0b" (UID: "50a0f50e-5f3a-488e-a1f9-0501fe243d0b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:15:44.588513 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.588495 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:15:44.590338 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.590317 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-kube-api-access-zbsf6" (OuterVolumeSpecName: "kube-api-access-zbsf6") pod "50a0f50e-5f3a-488e-a1f9-0501fe243d0b" (UID: "50a0f50e-5f3a-488e-a1f9-0501fe243d0b"). InnerVolumeSpecName "kube-api-access-zbsf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:15:44.689526 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.689497 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbsf6\" (UniqueName: \"kubernetes.io/projected/50a0f50e-5f3a-488e-a1f9-0501fe243d0b-kube-api-access-zbsf6\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:15:44.809988 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.809947 2543 generic.go:356] "Generic (PLEG): container finished" podID="50a0f50e-5f3a-488e-a1f9-0501fe243d0b" containerID="f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d" exitCode=0 Jul 24 16:15:44.810181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.810004 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" Jul 24 16:15:44.810181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.810030 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" event={"ID":"50a0f50e-5f3a-488e-a1f9-0501fe243d0b","Type":"ContainerDied","Data":"f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d"} Jul 24 16:15:44.810181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.810095 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q" event={"ID":"50a0f50e-5f3a-488e-a1f9-0501fe243d0b","Type":"ContainerDied","Data":"55dd4e57c2350139426d2392436c7ae757ff2581ca8d7a6d09ea9528d7fe106b"} Jul 24 16:15:44.810181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.810112 2543 scope.go:117] "RemoveContainer" containerID="f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d" Jul 24 16:15:44.819777 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.819749 2543 scope.go:117] "RemoveContainer" containerID="f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d" Jul 24 16:15:44.820067 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:15:44.820045 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d\": container with ID starting with f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d not found: ID does not exist" containerID="f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d" Jul 24 16:15:44.820137 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.820077 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d"} err="failed to get container status \"f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d\": rpc error: code = NotFound desc = could not find container \"f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d\": container with ID starting with f453344f6cbccfed9b92a2354bd124313f804a8935b76f4da65fd0de4449004d not found: ID does not exist" Jul 24 16:15:44.836779 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.836755 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q"] Jul 24 16:15:44.840130 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:44.840110 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-b89f991b-85fc458466-nv92q"] Jul 24 16:15:45.044111 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:15:45.044083 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="50a0f50e-5f3a-488e-a1f9-0501fe243d0b" path="/var/lib/kubelet/pods/50a0f50e-5f3a-488e-a1f9-0501fe243d0b/volumes" Jul 24 16:16:09.146854 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:09.146823 2543 scope.go:117] "RemoveContainer" containerID="b458b8ad36b51626cc95eb6ba8f76b037a0ece179bb211aedd5ca9aed345a760" Jul 24 16:16:39.773766 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:39.773734 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz"] Jul 24 16:16:39.774376 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:39.774016 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" containerID="cri-o://74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa" gracePeriod=30 Jul 24 16:16:45.033768 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.033709 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:16:45.134183 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.134150 2543 generic.go:356] "Generic (PLEG): container finished" podID="db348170-ad5a-4c89-8a2f-0643c03955da" containerID="74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa" exitCode=0 Jul 24 16:16:45.134342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.134205 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" Jul 24 16:16:45.134342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.134238 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" event={"ID":"db348170-ad5a-4c89-8a2f-0643c03955da","Type":"ContainerDied","Data":"74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa"} Jul 24 16:16:45.134342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.134295 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz" event={"ID":"db348170-ad5a-4c89-8a2f-0643c03955da","Type":"ContainerDied","Data":"39995721987d82e204bf0d315b8c5cd1bc4a28648c57b3fd85be71afb01c83f2"} Jul 24 16:16:45.134342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.134325 2543 scope.go:117] "RemoveContainer" containerID="74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa" Jul 24 16:16:45.143587 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.143545 2543 scope.go:117] "RemoveContainer" containerID="aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a" Jul 24 16:16:45.151602 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.151583 2543 scope.go:117] "RemoveContainer" containerID="74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa" Jul 24 16:16:45.151868 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:16:45.151849 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa\": container with ID starting with 74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa not found: ID does not exist" containerID="74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa" Jul 24 16:16:45.151922 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.151877 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa"} err="failed to get container status \"74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa\": rpc error: code = NotFound desc = could not find container \"74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa\": container with ID starting with 74eeb26cd8a794831a426e06b858478ba5d987703b3b32d5e02e2344359f70aa not found: ID does not exist" Jul 24 16:16:45.151922 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.151894 2543 scope.go:117] "RemoveContainer" containerID="aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a" Jul 24 16:16:45.152151 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:16:45.152134 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a\": container with ID starting with aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a not found: ID does not exist" containerID="aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a" Jul 24 16:16:45.152198 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.152159 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a"} err="failed to get container status \"aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a\": rpc error: code = NotFound desc = could not find container \"aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a\": container with ID starting with aa17de429f12583542e93b9ccc0cf97a6a999144d6fb1312a49b99fc4556247a not found: ID does not exist" Jul 24 16:16:45.199602 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199580 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db348170-ad5a-4c89-8a2f-0643c03955da-istiod-ca-cert\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.199714 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199643 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-data\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.199714 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199683 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-credential-socket\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.199829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199720 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db348170-ad5a-4c89-8a2f-0643c03955da-istio-podinfo\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.199829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199743 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2qgj2\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-kube-api-access-2qgj2\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.199829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199764 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-socket\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.199829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199801 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-certs\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.200027 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199858 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-istio-token\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.200027 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199883 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-envoy\") pod \"db348170-ad5a-4c89-8a2f-0643c03955da\" (UID: \"db348170-ad5a-4c89-8a2f-0643c03955da\") " Jul 24 16:16:45.200130 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199987 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:16:45.200130 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.200063 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:16:45.200130 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.199991 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db348170-ad5a-4c89-8a2f-0643c03955da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:16:45.200289 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.200220 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:16:45.200342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.200291 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db348170-ad5a-4c89-8a2f-0643c03955da-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.200342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.200310 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.200342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.200325 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.201045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.200749 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-data" (OuterVolumeSpecName: "istio-data") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:16:45.202206 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.202171 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db348170-ad5a-4c89-8a2f-0643c03955da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:16:45.202206 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.202183 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-istio-token" (OuterVolumeSpecName: "istio-token") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:16:45.202355 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.202289 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:16:45.202404 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.202383 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-kube-api-access-2qgj2" (OuterVolumeSpecName: "kube-api-access-2qgj2") pod "db348170-ad5a-4c89-8a2f-0643c03955da" (UID: "db348170-ad5a-4c89-8a2f-0643c03955da"). InnerVolumeSpecName "kube-api-access-2qgj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:16:45.301926 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.301841 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.301926 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.301867 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.301926 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.301875 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.301926 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.301882 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db348170-ad5a-4c89-8a2f-0643c03955da-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.301926 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.301893 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qgj2\" (UniqueName: \"kubernetes.io/projected/db348170-ad5a-4c89-8a2f-0643c03955da-kube-api-access-2qgj2\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.301926 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.301902 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db348170-ad5a-4c89-8a2f-0643c03955da-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:16:45.456845 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.456807 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz"] Jul 24 16:16:45.462637 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:45.462614 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b89f991b-openshift-default-789d6444f6-8tvxz"] Jul 24 16:16:47.043644 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:16:47.043613 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" path="/var/lib/kubelet/pods/db348170-ad5a-4c89-8a2f-0643c03955da/volumes" Jul 24 16:17:09.170730 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:09.170694 2543 scope.go:117] "RemoveContainer" containerID="2911285c5a1d66d9db5ee68ca6b3a371277e6d5b3073f066bcfd833d45d319b6" Jul 24 16:17:09.179329 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:09.179310 2543 scope.go:117] "RemoveContainer" containerID="fe19961e61bcc0fc206ba99525b4983cc1d315c40072b3c4f26d920508d7678b" Jul 24 16:17:35.293803 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.293763 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq"] Jul 24 16:17:35.294288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294228 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="50a0f50e-5f3a-488e-a1f9-0501fe243d0b" containerName="payload-processing" Jul 24 16:17:35.294288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294242 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="50a0f50e-5f3a-488e-a1f9-0501fe243d0b" containerName="payload-processing" Jul 24 16:17:35.294288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294254 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" Jul 24 16:17:35.294288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294260 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" Jul 24 16:17:35.294288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294267 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" Jul 24 16:17:35.294288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294275 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" Jul 24 16:17:35.294525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294346 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="50a0f50e-5f3a-488e-a1f9-0501fe243d0b" containerName="payload-processing" Jul 24 16:17:35.294525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294354 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" Jul 24 16:17:35.294525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.294365 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="db348170-ad5a-4c89-8a2f-0643c03955da" containerName="istio-proxy" Jul 24 16:17:35.297753 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.297737 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.300452 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.300431 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-35cc2d-openshift-default-dockercfg-gn6gx\"" Jul 24 16:17:35.314374 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.314351 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq"] Jul 24 16:17:35.399113 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399078 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-istio-token\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399309 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399117 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a6f505-4542-448d-903e-4542fe2f040e-istiod-ca-cert\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399309 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399169 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a6f505-4542-448d-903e-4542fe2f040e-istio-podinfo\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399309 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399232 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-envoy\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399309 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399268 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq755\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-kube-api-access-cq755\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399309 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399304 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-data\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399536 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399419 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-socket\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399536 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399526 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-certs\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.399636 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.399553 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-credential-socket\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.500619 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500583 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-data\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.500824 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500631 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-socket\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.500824 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500691 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-certs\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.500824 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500712 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-credential-socket\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.500824 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500763 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-istio-token\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.500824 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500785 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a6f505-4542-448d-903e-4542fe2f040e-istiod-ca-cert\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500832 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a6f505-4542-448d-903e-4542fe2f040e-istio-podinfo\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500863 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-envoy\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500892 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq755\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-kube-api-access-cq755\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500923 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-data\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.500998 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-socket\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.501039 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-certs\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.501069 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-credential-socket\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.501619 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.501592 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a6f505-4542-448d-903e-4542fe2f040e-istiod-ca-cert\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.503224 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.503196 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-envoy\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.503347 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.503328 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a6f505-4542-448d-903e-4542fe2f040e-istio-podinfo\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.510380 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.510349 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq755\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-kube-api-access-cq755\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.510628 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.510607 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-istio-token\") pod \"e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.608142 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.608060 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:35.752617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.752592 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq"] Jul 24 16:17:35.754542 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:17:35.754513 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a6f505_4542_448d_903e_4542fe2f040e.slice/crio-fb83bb0112a80dd902a842945ee68e60c8048c2e8da2a5ab508e56123995c7a6 WatchSource:0}: Error finding container fb83bb0112a80dd902a842945ee68e60c8048c2e8da2a5ab508e56123995c7a6: Status 404 returned error can't find the container with id fb83bb0112a80dd902a842945ee68e60c8048c2e8da2a5ab508e56123995c7a6 Jul 24 16:17:35.756641 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.756612 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:35.756713 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.756673 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:35.756713 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:35.756701 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:36.414979 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:36.414943 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" event={"ID":"26a6f505-4542-448d-903e-4542fe2f040e","Type":"ContainerStarted","Data":"86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560"} Jul 24 16:17:36.415376 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:36.415007 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" event={"ID":"26a6f505-4542-448d-903e-4542fe2f040e","Type":"ContainerStarted","Data":"fb83bb0112a80dd902a842945ee68e60c8048c2e8da2a5ab508e56123995c7a6"} Jul 24 16:17:36.438109 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:36.438052 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podStartSLOduration=1.43803451 podStartE2EDuration="1.43803451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:36.435806833 +0000 UTC m=+2187.967394445" watchObservedRunningTime="2026-07-24 16:17:36.43803451 +0000 UTC m=+2187.969622122" Jul 24 16:17:36.608526 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:36.608485 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:36.614386 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:36.614357 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:37.421141 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:37.421113 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:37.422012 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:37.421992 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:39.309771 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.309736 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g"] Jul 24 16:17:39.313671 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.313647 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.316158 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.316130 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-35cc2d\"" Jul 24 16:17:39.316323 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.316228 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-35cc2d-dockercfg-v4dr9\"" Jul 24 16:17:39.325764 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.325742 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g"] Jul 24 16:17:39.443435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.443396 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d262d4d-f7bb-4186-a15a-92e12870ac49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.443644 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.443529 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg8tt\" (UniqueName: \"kubernetes.io/projected/3d262d4d-f7bb-4186-a15a-92e12870ac49-kube-api-access-fg8tt\") pod \"payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.544183 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.544146 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d262d4d-f7bb-4186-a15a-92e12870ac49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.544366 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.544250 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg8tt\" (UniqueName: \"kubernetes.io/projected/3d262d4d-f7bb-4186-a15a-92e12870ac49-kube-api-access-fg8tt\") pod \"payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.545062 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.545035 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d262d4d-f7bb-4186-a15a-92e12870ac49-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.552702 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.552679 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg8tt\" (UniqueName: \"kubernetes.io/projected/3d262d4d-f7bb-4186-a15a-92e12870ac49-kube-api-access-fg8tt\") pod \"payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.627480 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.627400 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:39.777069 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:39.777042 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g"] Jul 24 16:17:39.778079 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:17:39.778056 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d262d4d_f7bb_4186_a15a_92e12870ac49.slice/crio-1332e2b4e39384843fc1a2cb3395c5a65d56350f3a92d06c3aad9cb094b55723 WatchSource:0}: Error finding container 1332e2b4e39384843fc1a2cb3395c5a65d56350f3a92d06c3aad9cb094b55723: Status 404 returned error can't find the container with id 1332e2b4e39384843fc1a2cb3395c5a65d56350f3a92d06c3aad9cb094b55723 Jul 24 16:17:40.438536 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:40.438500 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" event={"ID":"3d262d4d-f7bb-4186-a15a-92e12870ac49","Type":"ContainerStarted","Data":"de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07"} Jul 24 16:17:40.439076 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:40.438586 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" event={"ID":"3d262d4d-f7bb-4186-a15a-92e12870ac49","Type":"ContainerStarted","Data":"1332e2b4e39384843fc1a2cb3395c5a65d56350f3a92d06c3aad9cb094b55723"} Jul 24 16:17:40.439076 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:40.438716 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:40.457982 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:40.457937 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" podStartSLOduration=1.45792264 podStartE2EDuration="1.45792264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:40.456908855 +0000 UTC m=+2191.988496481" watchObservedRunningTime="2026-07-24 16:17:40.45792264 +0000 UTC m=+2191.989510253" Jul 24 16:17:43.503009 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.502973 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2"] Jul 24 16:17:43.507070 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.507046 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.509477 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.509451 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-47a0d8-openshift-default-dockercfg-6x9gt\"" Jul 24 16:17:43.519610 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.519549 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2"] Jul 24 16:17:43.584232 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584196 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-token\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584260 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq5r9\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-kube-api-access-sq5r9\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584324 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-data\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584350 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-socket\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584371 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-credential-socket\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584389 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-certs\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584712 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584440 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7aad3-6951-4833-b748-338ab02f7be5-istiod-ca-cert\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584712 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584483 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-podinfo\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.584712 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.584521 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-envoy\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685242 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685198 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-data\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685431 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685257 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-socket\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685431 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685300 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-credential-socket\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685431 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685329 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-certs\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685431 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685360 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7aad3-6951-4833-b748-338ab02f7be5-istiod-ca-cert\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685431 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685396 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-podinfo\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685722 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685452 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-envoy\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685722 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685525 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-token\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685722 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685623 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq5r9\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-kube-api-access-sq5r9\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685722 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685709 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-data\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.685915 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.685797 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-credential-socket\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.686041 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.686008 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-socket\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.686126 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.686042 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-certs\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.686358 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.686337 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7aad3-6951-4833-b748-338ab02f7be5-istiod-ca-cert\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.688330 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.688297 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-envoy\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.688521 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.688504 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-podinfo\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.693455 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.693430 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-token\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.693708 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.693689 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq5r9\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-kube-api-access-sq5r9\") pod \"e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.818870 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.818792 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:43.965571 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.965535 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2"] Jul 24 16:17:43.967268 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:17:43.967236 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3f7aad3_6951_4833_b748_338ab02f7be5.slice/crio-9e668b51887ea12c70cbb907016eb0e486f8d069c140b47be553fa446039678a WatchSource:0}: Error finding container 9e668b51887ea12c70cbb907016eb0e486f8d069c140b47be553fa446039678a: Status 404 returned error can't find the container with id 9e668b51887ea12c70cbb907016eb0e486f8d069c140b47be553fa446039678a Jul 24 16:17:43.969708 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.969666 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:43.969795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.969760 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:43.969836 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:43.969805 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:44.464351 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:44.464315 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" event={"ID":"e3f7aad3-6951-4833-b748-338ab02f7be5","Type":"ContainerStarted","Data":"c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952"} Jul 24 16:17:44.464514 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:44.464361 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" event={"ID":"e3f7aad3-6951-4833-b748-338ab02f7be5","Type":"ContainerStarted","Data":"9e668b51887ea12c70cbb907016eb0e486f8d069c140b47be553fa446039678a"} Jul 24 16:17:44.485217 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:44.485169 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podStartSLOduration=1.4851542549999999 podStartE2EDuration="1.485154255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:44.484144468 +0000 UTC m=+2196.015732082" watchObservedRunningTime="2026-07-24 16:17:44.485154255 +0000 UTC m=+2196.016741867" Jul 24 16:17:44.819724 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:44.819692 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:45.829783 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:45.829743 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:46.475806 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:46.475779 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:46.476689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:46.476666 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:17:47.420480 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.420436 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf"] Jul 24 16:17:47.425738 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.425710 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.434585 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.431324 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-47a0d8\"" Jul 24 16:17:47.434585 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.431602 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-47a0d8-dockercfg-pp8b7\"" Jul 24 16:17:47.444306 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.443635 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf"] Jul 24 16:17:47.533541 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.533478 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7e497-9265-4e54-aeb0-10e106222b98-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.533760 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.533609 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfpx4\" (UniqueName: \"kubernetes.io/projected/7ff7e497-9265-4e54-aeb0-10e106222b98-kube-api-access-nfpx4\") pod \"payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.635186 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.635143 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7e497-9265-4e54-aeb0-10e106222b98-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.635403 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.635228 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfpx4\" (UniqueName: \"kubernetes.io/projected/7ff7e497-9265-4e54-aeb0-10e106222b98-kube-api-access-nfpx4\") pod \"payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.637222 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.637193 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7e497-9265-4e54-aeb0-10e106222b98-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.645903 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.645878 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfpx4\" (UniqueName: \"kubernetes.io/projected/7ff7e497-9265-4e54-aeb0-10e106222b98-kube-api-access-nfpx4\") pod \"payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.749942 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.749904 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:47.954133 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:47.954105 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf"] Jul 24 16:17:48.489824 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:48.489784 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" event={"ID":"7ff7e497-9265-4e54-aeb0-10e106222b98","Type":"ContainerStarted","Data":"0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379"} Jul 24 16:17:48.490210 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:48.489852 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" event={"ID":"7ff7e497-9265-4e54-aeb0-10e106222b98","Type":"ContainerStarted","Data":"a58d6442268a36fb69525192492dfef78076e69360db3b9fdf968440250578f6"} Jul 24 16:17:48.490210 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:48.490091 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:17:48.510969 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:48.510915 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" podStartSLOduration=1.510895386 podStartE2EDuration="1.510895386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:17:48.507782641 +0000 UTC m=+2200.039370255" watchObservedRunningTime="2026-07-24 16:17:48.510895386 +0000 UTC m=+2200.042483002" Jul 24 16:17:51.445548 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:51.445516 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:17:52.528832 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:52.528793 2543 patch_prober.go:28] interesting pod/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:44114->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 24 16:17:52.529286 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:52.528869 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:44114->10.134.0.56:15021: read: connection reset by peer" Jul 24 16:17:53.528171 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:53.528131 2543 generic.go:356] "Generic (PLEG): container finished" podID="26a6f505-4542-448d-903e-4542fe2f040e" containerID="86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560" exitCode=0 Jul 24 16:17:53.528369 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:53.528201 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" event={"ID":"26a6f505-4542-448d-903e-4542fe2f040e","Type":"ContainerDied","Data":"86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560"} Jul 24 16:17:53.528751 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:53.528724 2543 scope.go:117] "RemoveContainer" containerID="86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560" Jul 24 16:17:53.530454 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:53.530419 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:53.531295 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:53.531268 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:53.531474 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:53.531453 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:17:54.542727 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:54.542684 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" event={"ID":"26a6f505-4542-448d-903e-4542fe2f040e","Type":"ContainerStarted","Data":"413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3"} Jul 24 16:17:54.616592 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:54.615753 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:54.616592 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:54.616310 2543 patch_prober.go:28] interesting pod/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 16:17:54.616592 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:54.616370 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 16:17:55.609165 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:55.609130 2543 patch_prober.go:28] interesting pod/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 16:17:55.609767 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:55.609732 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 16:17:56.608948 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:56.608901 2543 patch_prober.go:28] interesting pod/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 16:17:56.609170 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:56.608967 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 16:17:57.609331 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:57.609293 2543 patch_prober.go:28] interesting pod/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 24 16:17:57.609939 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:57.609903 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 24 16:17:58.613131 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:58.613098 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:58.613617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:58.613488 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:58.614212 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:58.614191 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:17:59.497432 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:17:59.497401 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:18:00.585259 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:00.585227 2543 generic.go:356] "Generic (PLEG): container finished" podID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerID="c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952" exitCode=0 Jul 24 16:18:00.585700 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:00.585301 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" event={"ID":"e3f7aad3-6951-4833-b748-338ab02f7be5","Type":"ContainerDied","Data":"c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952"} Jul 24 16:18:00.585764 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:00.585725 2543 scope.go:117] "RemoveContainer" containerID="c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952" Jul 24 16:18:00.586714 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:00.586686 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:18:00.586831 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:00.586766 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:18:00.586831 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:00.586824 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:18:01.476234 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:01.476206 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:18:01.593912 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:01.593871 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" event={"ID":"e3f7aad3-6951-4833-b748-338ab02f7be5","Type":"ContainerStarted","Data":"b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67"} Jul 24 16:18:01.819879 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:01.819771 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:18:01.820954 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:01.820920 2543 patch_prober.go:28] interesting pod/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 16:18:01.821099 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:01.821012 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 16:18:02.820108 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:02.820064 2543 patch_prober.go:28] interesting pod/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 16:18:02.820623 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:02.820133 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 16:18:03.819906 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:03.819858 2543 patch_prober.go:28] interesting pod/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 16:18:03.820128 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:03.819948 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 16:18:04.819524 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:04.819487 2543 patch_prober.go:28] interesting pod/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 24 16:18:04.819848 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:04.819811 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 24 16:18:05.823874 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:05.823843 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:18:05.824253 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:05.824087 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:18:05.824737 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:05.824713 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:18:09.211829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:18:09.211795 2543 scope.go:117] "RemoveContainer" containerID="40ebaed8e9f739103d4cf0b32d767d9a2d89845a58a52a9ba0913f66b82f3e7a" Jul 24 16:19:17.251949 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.251912 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g"] Jul 24 16:19:17.252489 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.252205 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" podUID="3d262d4d-f7bb-4186-a15a-92e12870ac49" containerName="payload-processing" containerID="cri-o://de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07" gracePeriod=30 Jul 24 16:19:17.528644 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.528617 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:19:17.576173 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.576136 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d262d4d-f7bb-4186-a15a-92e12870ac49-plugins-config-volume\") pod \"3d262d4d-f7bb-4186-a15a-92e12870ac49\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " Jul 24 16:19:17.576381 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.576245 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg8tt\" (UniqueName: \"kubernetes.io/projected/3d262d4d-f7bb-4186-a15a-92e12870ac49-kube-api-access-fg8tt\") pod \"3d262d4d-f7bb-4186-a15a-92e12870ac49\" (UID: \"3d262d4d-f7bb-4186-a15a-92e12870ac49\") " Jul 24 16:19:17.576659 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.576556 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d262d4d-f7bb-4186-a15a-92e12870ac49-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d262d4d-f7bb-4186-a15a-92e12870ac49" (UID: "3d262d4d-f7bb-4186-a15a-92e12870ac49"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:19:17.578509 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.578484 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d262d4d-f7bb-4186-a15a-92e12870ac49-kube-api-access-fg8tt" (OuterVolumeSpecName: "kube-api-access-fg8tt") pod "3d262d4d-f7bb-4186-a15a-92e12870ac49" (UID: "3d262d4d-f7bb-4186-a15a-92e12870ac49"). InnerVolumeSpecName "kube-api-access-fg8tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:17.677417 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.677384 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d262d4d-f7bb-4186-a15a-92e12870ac49-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:17.677417 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:17.677413 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg8tt\" (UniqueName: \"kubernetes.io/projected/3d262d4d-f7bb-4186-a15a-92e12870ac49-kube-api-access-fg8tt\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:18.011746 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.011707 2543 generic.go:356] "Generic (PLEG): container finished" podID="3d262d4d-f7bb-4186-a15a-92e12870ac49" containerID="de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07" exitCode=0 Jul 24 16:19:18.011933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.011758 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" Jul 24 16:19:18.011933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.011800 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" event={"ID":"3d262d4d-f7bb-4186-a15a-92e12870ac49","Type":"ContainerDied","Data":"de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07"} Jul 24 16:19:18.011933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.011876 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g" event={"ID":"3d262d4d-f7bb-4186-a15a-92e12870ac49","Type":"ContainerDied","Data":"1332e2b4e39384843fc1a2cb3395c5a65d56350f3a92d06c3aad9cb094b55723"} Jul 24 16:19:18.011933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.011912 2543 scope.go:117] "RemoveContainer" containerID="de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07" Jul 24 16:19:18.024122 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.024070 2543 scope.go:117] "RemoveContainer" containerID="de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07" Jul 24 16:19:18.024533 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:19:18.024509 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07\": container with ID starting with de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07 not found: ID does not exist" containerID="de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07" Jul 24 16:19:18.024841 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.024545 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07"} err="failed to get container status \"de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07\": rpc error: code = NotFound desc = could not find container \"de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07\": container with ID starting with de3535049ab8e53fd87991e3daf631a34f4a8b9935ce47755dd36757ff42fb07 not found: ID does not exist" Jul 24 16:19:18.047254 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.047219 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g"] Jul 24 16:19:18.053527 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:18.053492 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-35cc2d-db4f58fc4-hgc7g"] Jul 24 16:19:19.043511 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:19.043479 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d262d4d-f7bb-4186-a15a-92e12870ac49" path="/var/lib/kubelet/pods/3d262d4d-f7bb-4186-a15a-92e12870ac49/volumes" Jul 24 16:19:28.632980 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:28.632944 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq"] Jul 24 16:19:28.633466 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:28.633212 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" containerID="cri-o://413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3" gracePeriod=30 Jul 24 16:19:33.905362 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:33.905303 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:19:34.043603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043535 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a6f505-4542-448d-903e-4542fe2f040e-istiod-ca-cert\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043604 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a6f505-4542-448d-903e-4542fe2f040e-istio-podinfo\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043835 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043634 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq755\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-kube-api-access-cq755\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043835 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043677 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-istio-token\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043835 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043723 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-envoy\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043835 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043764 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-socket\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043835 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043801 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-certs\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.043835 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043824 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-credential-socket\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.044159 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.043891 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-data\") pod \"26a6f505-4542-448d-903e-4542fe2f040e\" (UID: \"26a6f505-4542-448d-903e-4542fe2f040e\") " Jul 24 16:19:34.044159 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044073 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:34.044159 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044108 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26a6f505-4542-448d-903e-4542fe2f040e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:19:34.044324 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044240 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:34.044383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044354 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/26a6f505-4542-448d-903e-4542fe2f040e-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.044437 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044397 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.044437 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044414 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.044758 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044715 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:34.045020 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.044780 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-data" (OuterVolumeSpecName: "istio-data") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:34.046030 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.046002 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/26a6f505-4542-448d-903e-4542fe2f040e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:19:34.046175 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.046029 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-kube-api-access-cq755" (OuterVolumeSpecName: "kube-api-access-cq755") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "kube-api-access-cq755". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:34.046311 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.046298 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:34.046437 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.046423 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-istio-token" (OuterVolumeSpecName: "istio-token") pod "26a6f505-4542-448d-903e-4542fe2f040e" (UID: "26a6f505-4542-448d-903e-4542fe2f040e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:34.100483 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.100449 2543 generic.go:356] "Generic (PLEG): container finished" podID="26a6f505-4542-448d-903e-4542fe2f040e" containerID="413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3" exitCode=0 Jul 24 16:19:34.100692 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.100507 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" Jul 24 16:19:34.100692 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.100539 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" event={"ID":"26a6f505-4542-448d-903e-4542fe2f040e","Type":"ContainerDied","Data":"413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3"} Jul 24 16:19:34.100692 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.100653 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq" event={"ID":"26a6f505-4542-448d-903e-4542fe2f040e","Type":"ContainerDied","Data":"fb83bb0112a80dd902a842945ee68e60c8048c2e8da2a5ab508e56123995c7a6"} Jul 24 16:19:34.100692 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.100683 2543 scope.go:117] "RemoveContainer" containerID="413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3" Jul 24 16:19:34.110807 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.110792 2543 scope.go:117] "RemoveContainer" containerID="86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560" Jul 24 16:19:34.119030 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.119015 2543 scope.go:117] "RemoveContainer" containerID="413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3" Jul 24 16:19:34.119276 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:19:34.119257 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3\": container with ID starting with 413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3 not found: ID does not exist" containerID="413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3" Jul 24 16:19:34.119339 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.119284 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3"} err="failed to get container status \"413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3\": rpc error: code = NotFound desc = could not find container \"413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3\": container with ID starting with 413420657cc1f6e49d09ce2f13da89d18fe558eb7a891e18042918c52261b1e3 not found: ID does not exist" Jul 24 16:19:34.119339 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.119300 2543 scope.go:117] "RemoveContainer" containerID="86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560" Jul 24 16:19:34.119547 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:19:34.119529 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560\": container with ID starting with 86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560 not found: ID does not exist" containerID="86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560" Jul 24 16:19:34.119617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.119553 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560"} err="failed to get container status \"86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560\": rpc error: code = NotFound desc = could not find container \"86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560\": container with ID starting with 86123b28de44d110599d6ed91c7f7bb97a07b274623792fee8b012ff5ae02560 not found: ID does not exist" Jul 24 16:19:34.124132 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.124113 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq"] Jul 24 16:19:34.127858 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.127829 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-35cc2d-openshift-default-79658cdc78-nmdlq"] Jul 24 16:19:34.145225 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.145201 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/26a6f505-4542-448d-903e-4542fe2f040e-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.145225 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.145223 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq755\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-kube-api-access-cq755\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.145349 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.145234 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/26a6f505-4542-448d-903e-4542fe2f040e-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.145349 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.145243 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.145349 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.145251 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:34.145349 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:34.145259 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/26a6f505-4542-448d-903e-4542fe2f040e-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:35.044434 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:35.044400 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26a6f505-4542-448d-903e-4542fe2f040e" path="/var/lib/kubelet/pods/26a6f505-4542-448d-903e-4542fe2f040e/volumes" Jul 24 16:19:39.453621 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.453549 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf"] Jul 24 16:19:39.454089 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.453911 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" podUID="7ff7e497-9265-4e54-aeb0-10e106222b98" containerName="payload-processing" containerID="cri-o://0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379" gracePeriod=30 Jul 24 16:19:39.496595 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.496534 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" podUID="7ff7e497-9265-4e54-aeb0-10e106222b98" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.59:9004: connect: connection refused" Jul 24 16:19:39.745768 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.745743 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:19:39.801061 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.801027 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfpx4\" (UniqueName: \"kubernetes.io/projected/7ff7e497-9265-4e54-aeb0-10e106222b98-kube-api-access-nfpx4\") pod \"7ff7e497-9265-4e54-aeb0-10e106222b98\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " Jul 24 16:19:39.801259 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.801141 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7e497-9265-4e54-aeb0-10e106222b98-plugins-config-volume\") pod \"7ff7e497-9265-4e54-aeb0-10e106222b98\" (UID: \"7ff7e497-9265-4e54-aeb0-10e106222b98\") " Jul 24 16:19:39.801498 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.801472 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff7e497-9265-4e54-aeb0-10e106222b98-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7ff7e497-9265-4e54-aeb0-10e106222b98" (UID: "7ff7e497-9265-4e54-aeb0-10e106222b98"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:19:39.803319 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.803295 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff7e497-9265-4e54-aeb0-10e106222b98-kube-api-access-nfpx4" (OuterVolumeSpecName: "kube-api-access-nfpx4") pod "7ff7e497-9265-4e54-aeb0-10e106222b98" (UID: "7ff7e497-9265-4e54-aeb0-10e106222b98"). InnerVolumeSpecName "kube-api-access-nfpx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:39.902234 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.902202 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfpx4\" (UniqueName: \"kubernetes.io/projected/7ff7e497-9265-4e54-aeb0-10e106222b98-kube-api-access-nfpx4\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:39.902234 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:39.902229 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ff7e497-9265-4e54-aeb0-10e106222b98-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:40.133435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.133349 2543 generic.go:356] "Generic (PLEG): container finished" podID="7ff7e497-9265-4e54-aeb0-10e106222b98" containerID="0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379" exitCode=0 Jul 24 16:19:40.133435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.133391 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" Jul 24 16:19:40.133710 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.133439 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" event={"ID":"7ff7e497-9265-4e54-aeb0-10e106222b98","Type":"ContainerDied","Data":"0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379"} Jul 24 16:19:40.133710 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.133498 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf" event={"ID":"7ff7e497-9265-4e54-aeb0-10e106222b98","Type":"ContainerDied","Data":"a58d6442268a36fb69525192492dfef78076e69360db3b9fdf968440250578f6"} Jul 24 16:19:40.133710 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.133530 2543 scope.go:117] "RemoveContainer" containerID="0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379" Jul 24 16:19:40.142708 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.142691 2543 scope.go:117] "RemoveContainer" containerID="0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379" Jul 24 16:19:40.142963 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:19:40.142946 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379\": container with ID starting with 0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379 not found: ID does not exist" containerID="0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379" Jul 24 16:19:40.143001 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.142975 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379"} err="failed to get container status \"0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379\": rpc error: code = NotFound desc = could not find container \"0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379\": container with ID starting with 0b7a19389fe6413f4ca07c001c5f8654151e7890c9b79c231a1998eb2c8c1379 not found: ID does not exist" Jul 24 16:19:40.155976 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.155904 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf"] Jul 24 16:19:40.159424 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:40.159399 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-47a0d8-7886dd4988-qrljf"] Jul 24 16:19:41.043870 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:41.043831 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ff7e497-9265-4e54-aeb0-10e106222b98" path="/var/lib/kubelet/pods/7ff7e497-9265-4e54-aeb0-10e106222b98/volumes" Jul 24 16:19:50.390936 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:50.390905 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2"] Jul 24 16:19:50.391426 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:50.391142 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" containerID="cri-o://b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67" gracePeriod=30 Jul 24 16:19:50.826284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:50.826248 2543 patch_prober.go:28] interesting pod/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 16:19:50.826455 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:50.826299 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 16:19:51.296955 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.296921 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5"] Jul 24 16:19:51.297351 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297338 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ff7e497-9265-4e54-aeb0-10e106222b98" containerName="payload-processing" Jul 24 16:19:51.297397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297353 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ff7e497-9265-4e54-aeb0-10e106222b98" containerName="payload-processing" Jul 24 16:19:51.297397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297370 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d262d4d-f7bb-4186-a15a-92e12870ac49" containerName="payload-processing" Jul 24 16:19:51.297397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297375 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d262d4d-f7bb-4186-a15a-92e12870ac49" containerName="payload-processing" Jul 24 16:19:51.297397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297383 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" Jul 24 16:19:51.297397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297388 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" Jul 24 16:19:51.297554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297472 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" Jul 24 16:19:51.297554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297483 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ff7e497-9265-4e54-aeb0-10e106222b98" containerName="payload-processing" Jul 24 16:19:51.297554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.297490 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d262d4d-f7bb-4186-a15a-92e12870ac49" containerName="payload-processing" Jul 24 16:19:51.300799 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.300781 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.303497 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.303474 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1d6db7-openshift-default-dockercfg-tqgww\"" Jul 24 16:19:51.314313 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.314294 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5"] Jul 24 16:19:51.422470 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422437 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-certs\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422476 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-envoy\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422596 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-data\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422617 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-credential-socket\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422637 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4dhh\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-kube-api-access-n4dhh\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422666 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-socket\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422738 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-podinfo\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422781 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istiod-ca-cert\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.422881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.422816 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-token\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523572 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523534 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-socket\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523759 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523617 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-podinfo\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523759 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523643 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istiod-ca-cert\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523759 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523684 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-token\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523759 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523735 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-certs\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523989 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523760 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-envoy\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.523989 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523909 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-data\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524100 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.523983 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-socket\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524100 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.524020 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-credential-socket\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524100 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.524057 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4dhh\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-kube-api-access-n4dhh\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524257 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.524241 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-certs\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524314 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.524240 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-data\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524364 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.524321 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-credential-socket\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.524424 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.524390 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istiod-ca-cert\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.526150 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.526124 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-envoy\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.526501 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.526479 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-podinfo\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.532229 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.532207 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-token\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.532351 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.532323 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4dhh\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-kube-api-access-n4dhh\") pod \"e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.614201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.614112 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:51.766625 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.766604 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5"] Jul 24 16:19:51.769194 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:19:51.769167 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b0c4ae9_1af1_4c66_ab1a_aacec7fb3ace.slice/crio-eb494b9dad1a62a496bfdc30083b88a7cbd5d21435f3d0e21cd8eba625f4a87a WatchSource:0}: Error finding container eb494b9dad1a62a496bfdc30083b88a7cbd5d21435f3d0e21cd8eba625f4a87a: Status 404 returned error can't find the container with id eb494b9dad1a62a496bfdc30083b88a7cbd5d21435f3d0e21cd8eba625f4a87a Jul 24 16:19:51.771285 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.771253 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:19:51.771368 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.771316 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:19:51.771368 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:51.771343 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:19:52.198870 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:52.198836 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" event={"ID":"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace","Type":"ContainerStarted","Data":"aeb8b82b1c6e7b915fa5065a361897fbecafd96291e46c3fea753610a69c3f33"} Jul 24 16:19:52.199044 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:52.198895 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" event={"ID":"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace","Type":"ContainerStarted","Data":"eb494b9dad1a62a496bfdc30083b88a7cbd5d21435f3d0e21cd8eba625f4a87a"} Jul 24 16:19:52.218475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:52.218416 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podStartSLOduration=1.218395006 podStartE2EDuration="1.218395006s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:19:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:19:52.21706228 +0000 UTC m=+2323.748649911" watchObservedRunningTime="2026-07-24 16:19:52.218395006 +0000 UTC m=+2323.749982620" Jul 24 16:19:52.615184 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:52.615155 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:52.620006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:52.619983 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:53.204988 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:53.204954 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:53.205925 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:53.205898 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:19:55.660614 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.660528 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:19:55.771161 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771127 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7aad3-6951-4833-b748-338ab02f7be5-istiod-ca-cert\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771175 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-envoy\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771228 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-socket\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771269 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sq5r9\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-kube-api-access-sq5r9\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771318 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-data\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771338 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-credential-socket\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771675 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771393 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-podinfo\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771675 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771420 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-certs\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771675 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771483 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-token\") pod \"e3f7aad3-6951-4833-b748-338ab02f7be5\" (UID: \"e3f7aad3-6951-4833-b748-338ab02f7be5\") " Jul 24 16:19:55.771675 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771484 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:55.771675 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771513 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3f7aad3-6951-4833-b748-338ab02f7be5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:19:55.771949 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771681 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:55.771949 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.771803 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:55.772287 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.772197 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.772287 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.772225 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.772287 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.772241 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.772287 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.772254 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3f7aad3-6951-4833-b748-338ab02f7be5-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.772670 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.772486 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-data" (OuterVolumeSpecName: "istio-data") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:55.773680 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.773654 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:19:55.773789 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.773676 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-kube-api-access-sq5r9" (OuterVolumeSpecName: "kube-api-access-sq5r9") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "kube-api-access-sq5r9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:55.773789 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.773681 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:19:55.773941 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.773925 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-token" (OuterVolumeSpecName: "istio-token") pod "e3f7aad3-6951-4833-b748-338ab02f7be5" (UID: "e3f7aad3-6951-4833-b748-338ab02f7be5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:19:55.873319 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.873290 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq5r9\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-kube-api-access-sq5r9\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.873319 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.873317 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.873319 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.873326 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.873603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.873333 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:55.873603 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:55.873341 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3f7aad3-6951-4833-b748-338ab02f7be5-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:19:56.223045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.223013 2543 generic.go:356] "Generic (PLEG): container finished" podID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerID="b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67" exitCode=0 Jul 24 16:19:56.223204 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.223066 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" Jul 24 16:19:56.223204 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.223104 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" event={"ID":"e3f7aad3-6951-4833-b748-338ab02f7be5","Type":"ContainerDied","Data":"b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67"} Jul 24 16:19:56.223204 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.223160 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2" event={"ID":"e3f7aad3-6951-4833-b748-338ab02f7be5","Type":"ContainerDied","Data":"9e668b51887ea12c70cbb907016eb0e486f8d069c140b47be553fa446039678a"} Jul 24 16:19:56.223204 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.223182 2543 scope.go:117] "RemoveContainer" containerID="b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67" Jul 24 16:19:56.236466 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.236445 2543 scope.go:117] "RemoveContainer" containerID="c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952" Jul 24 16:19:56.245314 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.245299 2543 scope.go:117] "RemoveContainer" containerID="b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67" Jul 24 16:19:56.245610 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:19:56.245587 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67\": container with ID starting with b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67 not found: ID does not exist" containerID="b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67" Jul 24 16:19:56.245690 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.245619 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67"} err="failed to get container status \"b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67\": rpc error: code = NotFound desc = could not find container \"b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67\": container with ID starting with b599844df9a792c32d88c4002cd1cc5e8420b8ec9deec0ca51061f9fde470a67 not found: ID does not exist" Jul 24 16:19:56.245690 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.245637 2543 scope.go:117] "RemoveContainer" containerID="c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952" Jul 24 16:19:56.245890 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:19:56.245865 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952\": container with ID starting with c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952 not found: ID does not exist" containerID="c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952" Jul 24 16:19:56.245945 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.245900 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952"} err="failed to get container status \"c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952\": rpc error: code = NotFound desc = could not find container \"c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952\": container with ID starting with c37a2c01c8b1d1b6c20f8197fc198399b19040bb6a0fe725a2077126581d7952 not found: ID does not exist" Jul 24 16:19:56.247108 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.247085 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2"] Jul 24 16:19:56.253368 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:56.253346 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-47a0d8-openshift-default-78cc79579b-mzpb2"] Jul 24 16:19:57.044509 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:57.044476 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" path="/var/lib/kubelet/pods/e3f7aad3-6951-4833-b748-338ab02f7be5/volumes" Jul 24 16:19:59.379363 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.379329 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq"] Jul 24 16:19:59.380097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.380072 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" Jul 24 16:19:59.380097 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.380097 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" Jul 24 16:19:59.380305 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.380139 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" Jul 24 16:19:59.380305 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.380148 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" Jul 24 16:19:59.380305 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.380250 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" Jul 24 16:19:59.380305 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.380260 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="26a6f505-4542-448d-903e-4542fe2f040e" containerName="istio-proxy" Jul 24 16:19:59.384977 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.384958 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.387256 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.387239 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e8e158-openshift-default-dockercfg-2cpt4\"" Jul 24 16:19:59.393702 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.393677 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq"] Jul 24 16:19:59.512864 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.512828 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-certs\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513056 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.512901 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-credential-socket\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513056 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.512965 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-token\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513056 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.513031 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-podinfo\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.513126 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9khz6\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-kube-api-access-9khz6\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.513160 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-data\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.513181 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-socket\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.513199 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-envoy\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.513332 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.513231 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5284567-ea1d-4658-b6a0-f13398624f1b-istiod-ca-cert\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614382 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614351 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-certs\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614595 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614425 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-credential-socket\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614595 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614458 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-token\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614595 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614486 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-podinfo\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614614 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9khz6\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-kube-api-access-9khz6\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614654 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-data\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614685 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-socket\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614715 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-envoy\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614736 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-certs\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.614765 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614750 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5284567-ea1d-4658-b6a0-f13398624f1b-istiod-ca-cert\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.615153 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.614948 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-credential-socket\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.615153 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.615096 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-socket\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.615271 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.615192 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-data\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.615380 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.615359 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5284567-ea1d-4658-b6a0-f13398624f1b-istiod-ca-cert\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.617130 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.617110 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-envoy\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.617665 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.617645 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-podinfo\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.622122 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.622105 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-token\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.622233 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.622219 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9khz6\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-kube-api-access-9khz6\") pod \"e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.696316 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.696229 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:19:59.844186 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.844158 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq"] Jul 24 16:19:59.846029 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:19:59.846004 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5284567_ea1d_4658_b6a0_f13398624f1b.slice/crio-4d97df42538f796da2a11122d751887ca88b1499a071a0ef524131dc29a22630 WatchSource:0}: Error finding container 4d97df42538f796da2a11122d751887ca88b1499a071a0ef524131dc29a22630: Status 404 returned error can't find the container with id 4d97df42538f796da2a11122d751887ca88b1499a071a0ef524131dc29a22630 Jul 24 16:19:59.848289 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.848259 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:19:59.848402 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.848333 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:19:59.848402 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:19:59.848378 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:00.248460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:00.248429 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" event={"ID":"d5284567-ea1d-4658-b6a0-f13398624f1b","Type":"ContainerStarted","Data":"c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117"} Jul 24 16:20:00.248672 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:00.248479 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" event={"ID":"d5284567-ea1d-4658-b6a0-f13398624f1b","Type":"ContainerStarted","Data":"4d97df42538f796da2a11122d751887ca88b1499a071a0ef524131dc29a22630"} Jul 24 16:20:00.268673 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:00.268614 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podStartSLOduration=1.268591568 podStartE2EDuration="1.268591568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:19:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:20:00.267749899 +0000 UTC m=+2331.799337531" watchObservedRunningTime="2026-07-24 16:20:00.268591568 +0000 UTC m=+2331.800179179" Jul 24 16:20:00.696862 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:00.696778 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:00.701949 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:00.701925 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:01.255741 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:01.255707 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:01.256802 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:01.256783 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:03.080000 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.079966 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj"] Jul 24 16:20:03.080410 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.080397 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" Jul 24 16:20:03.080454 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.080411 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" Jul 24 16:20:03.080490 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.080479 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3f7aad3-6951-4833-b748-338ab02f7be5" containerName="istio-proxy" Jul 24 16:20:03.083855 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.083828 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.086081 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.086053 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e8e158\"" Jul 24 16:20:03.086212 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.086135 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e8e158-dockercfg-ng85f\"" Jul 24 16:20:03.091254 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.091229 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj"] Jul 24 16:20:03.255645 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.255616 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdnl7\" (UniqueName: \"kubernetes.io/projected/818ac7a0-c51d-4a7a-9239-5447bf43477b-kube-api-access-jdnl7\") pod \"payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.255859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.255714 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818ac7a0-c51d-4a7a-9239-5447bf43477b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.356829 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.356747 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdnl7\" (UniqueName: \"kubernetes.io/projected/818ac7a0-c51d-4a7a-9239-5447bf43477b-kube-api-access-jdnl7\") pod \"payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.356995 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.356838 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818ac7a0-c51d-4a7a-9239-5447bf43477b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.357527 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.357507 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818ac7a0-c51d-4a7a-9239-5447bf43477b-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.364464 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.364443 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdnl7\" (UniqueName: \"kubernetes.io/projected/818ac7a0-c51d-4a7a-9239-5447bf43477b-kube-api-access-jdnl7\") pod \"payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.396472 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.396436 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:03.562852 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:03.562821 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj"] Jul 24 16:20:04.274393 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:04.274363 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" event={"ID":"818ac7a0-c51d-4a7a-9239-5447bf43477b","Type":"ContainerStarted","Data":"6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84"} Jul 24 16:20:04.274909 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:04.274445 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" event={"ID":"818ac7a0-c51d-4a7a-9239-5447bf43477b","Type":"ContainerStarted","Data":"3e3ee30941a0641cfc0ea3c77bee5fd7a2f0e88fc7bbc52679b8e30ff54c4046"} Jul 24 16:20:04.274909 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:04.274480 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:04.292902 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:04.292859 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" podStartSLOduration=1.292843595 podStartE2EDuration="1.292843595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:20:04.291383004 +0000 UTC m=+2335.822970619" watchObservedRunningTime="2026-07-24 16:20:04.292843595 +0000 UTC m=+2335.824431208" Jul 24 16:20:09.241064 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:09.241034 2543 scope.go:117] "RemoveContainer" containerID="2032b7c40645da96abd0548b3957d77dab3b233f6da67d270580b7d537a6b7fe" Jul 24 16:20:09.246742 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:09.246706 2543 patch_prober.go:28] interesting pod/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 16:20:09.246864 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:09.246756 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 16:20:11.317371 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:11.317334 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerID="aeb8b82b1c6e7b915fa5065a361897fbecafd96291e46c3fea753610a69c3f33" exitCode=0 Jul 24 16:20:11.317863 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:11.317408 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" event={"ID":"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace","Type":"ContainerDied","Data":"aeb8b82b1c6e7b915fa5065a361897fbecafd96291e46c3fea753610a69c3f33"} Jul 24 16:20:11.317863 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:11.317839 2543 scope.go:117] "RemoveContainer" containerID="aeb8b82b1c6e7b915fa5065a361897fbecafd96291e46c3fea753610a69c3f33" Jul 24 16:20:11.318868 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:11.318843 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:11.318929 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:11.318910 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:11.318965 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:11.318938 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:12.330116 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:12.330061 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" event={"ID":"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace","Type":"ContainerStarted","Data":"07db7f61b8792e1187949b4a5e91dd75881a685464c3770788a159623e87ebec"} Jul 24 16:20:12.615146 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:12.615041 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:20:12.615375 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:12.615346 2543 patch_prober.go:28] interesting pod/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 16:20:12.615468 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:12.615412 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 16:20:13.614903 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:13.614863 2543 patch_prober.go:28] interesting pod/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 16:20:13.615392 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:13.614931 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 16:20:14.615020 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:14.614967 2543 patch_prober.go:28] interesting pod/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 16:20:14.615707 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:14.615659 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 16:20:15.281969 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:15.281940 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:15.615645 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:15.614579 2543 patch_prober.go:28] interesting pod/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 24 16:20:15.615645 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:15.614644 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 24 16:20:16.618135 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:16.618056 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:20:16.618548 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:16.618531 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:20:16.619313 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:16.619293 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:20:19.389424 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:19.389390 2543 generic.go:356] "Generic (PLEG): container finished" podID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerID="c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117" exitCode=0 Jul 24 16:20:19.389892 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:19.389460 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" event={"ID":"d5284567-ea1d-4658-b6a0-f13398624f1b","Type":"ContainerDied","Data":"c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117"} Jul 24 16:20:19.389892 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:19.389884 2543 scope.go:117] "RemoveContainer" containerID="c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117" Jul 24 16:20:19.391073 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:19.391050 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:19.391166 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:19.391113 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:19.391166 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:19.391144 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:20:20.397452 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:20.397416 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" event={"ID":"d5284567-ea1d-4658-b6a0-f13398624f1b","Type":"ContainerStarted","Data":"6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5"} Jul 24 16:20:20.697072 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:20.696985 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:20.697234 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:20.697213 2543 patch_prober.go:28] interesting pod/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 16:20:20.697278 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:20.697262 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 16:20:21.697467 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:21.697425 2543 patch_prober.go:28] interesting pod/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 16:20:21.697925 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:21.697505 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 16:20:22.696951 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:22.696915 2543 patch_prober.go:28] interesting pod/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 16:20:22.697155 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:22.696976 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 16:20:23.697021 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:23.696983 2543 patch_prober.go:28] interesting pod/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 24 16:20:23.697417 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:23.697050 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 24 16:20:24.700612 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:24.700580 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:24.701002 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:24.700970 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:24.701764 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:24.701741 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:20:41.419218 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:41.419183 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5"] Jul 24 16:20:41.419819 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:41.419430 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" containerID="cri-o://07db7f61b8792e1187949b4a5e91dd75881a685464c3770788a159623e87ebec" gracePeriod=30 Jul 24 16:20:46.551166 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.551138 2543 generic.go:356] "Generic (PLEG): container finished" podID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerID="07db7f61b8792e1187949b4a5e91dd75881a685464c3770788a159623e87ebec" exitCode=0 Jul 24 16:20:46.551518 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.551218 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" event={"ID":"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace","Type":"ContainerDied","Data":"07db7f61b8792e1187949b4a5e91dd75881a685464c3770788a159623e87ebec"} Jul 24 16:20:46.551518 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.551297 2543 scope.go:117] "RemoveContainer" containerID="aeb8b82b1c6e7b915fa5065a361897fbecafd96291e46c3fea753610a69c3f33" Jul 24 16:20:46.690906 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.690848 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:20:46.806045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806016 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-socket\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.806244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806127 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istiod-ca-cert\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.806244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806158 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-data\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.806244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806182 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-podinfo\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.806244 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806206 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-credential-socket\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.806474 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806324 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-certs\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.806474 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806342 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806408 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-envoy\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806713 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-token\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806533 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806654 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.806753 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4dhh\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-kube-api-access-n4dhh\") pod \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\" (UID: \"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace\") " Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.807306 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.807327 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.807377 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.807341 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.807937 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.807524 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:20:46.807937 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.807726 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:20:46.808770 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.808747 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:20:46.808935 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.808918 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:20:46.809406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.809382 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:20:46.809488 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.809414 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-kube-api-access-n4dhh" (OuterVolumeSpecName: "kube-api-access-n4dhh") pod "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" (UID: "0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace"). InnerVolumeSpecName "kube-api-access-n4dhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:20:46.908359 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.908328 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.908359 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.908354 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.908359 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.908363 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4dhh\" (UniqueName: \"kubernetes.io/projected/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-kube-api-access-n4dhh\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.908610 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.908372 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.908610 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.908380 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:46.908610 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:46.908388 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:47.558550 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.558519 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" Jul 24 16:20:47.559023 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.558550 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" event={"ID":"0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace","Type":"ContainerDied","Data":"eb494b9dad1a62a496bfdc30083b88a7cbd5d21435f3d0e21cd8eba625f4a87a"} Jul 24 16:20:47.559023 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.558635 2543 scope.go:117] "RemoveContainer" containerID="07db7f61b8792e1187949b4a5e91dd75881a685464c3770788a159623e87ebec" Jul 24 16:20:47.581299 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.581246 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5"] Jul 24 16:20:47.584406 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.584376 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5"] Jul 24 16:20:47.619294 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.619262 2543 patch_prober.go:28] interesting pod/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 16:20:47.619433 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:47.619323 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-1d6db7-openshift-default-6b86dd986d-fq9k5" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 16:20:49.043733 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:49.043701 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" path="/var/lib/kubelet/pods/0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace/volumes" Jul 24 16:20:57.339581 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.339532 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj"] Jul 24 16:20:57.340035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.339816 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" podUID="818ac7a0-c51d-4a7a-9239-5447bf43477b" containerName="payload-pre-processing" containerID="cri-o://6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84" gracePeriod=30 Jul 24 16:20:57.609299 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.609274 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:57.616122 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.616095 2543 generic.go:356] "Generic (PLEG): container finished" podID="818ac7a0-c51d-4a7a-9239-5447bf43477b" containerID="6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84" exitCode=0 Jul 24 16:20:57.616289 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.616164 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" Jul 24 16:20:57.616289 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.616171 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" event={"ID":"818ac7a0-c51d-4a7a-9239-5447bf43477b","Type":"ContainerDied","Data":"6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84"} Jul 24 16:20:57.616410 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.616289 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj" event={"ID":"818ac7a0-c51d-4a7a-9239-5447bf43477b","Type":"ContainerDied","Data":"3e3ee30941a0641cfc0ea3c77bee5fd7a2f0e88fc7bbc52679b8e30ff54c4046"} Jul 24 16:20:57.616410 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.616308 2543 scope.go:117] "RemoveContainer" containerID="6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84" Jul 24 16:20:57.625179 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.625162 2543 scope.go:117] "RemoveContainer" containerID="6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84" Jul 24 16:20:57.625424 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:20:57.625403 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84\": container with ID starting with 6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84 not found: ID does not exist" containerID="6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84" Jul 24 16:20:57.625490 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.625438 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84"} err="failed to get container status \"6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84\": rpc error: code = NotFound desc = could not find container \"6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84\": container with ID starting with 6c00dcac19b8ef76786dd28121d2a7331a2c9aef89c9a01655fcd3d5c3ae0c84 not found: ID does not exist" Jul 24 16:20:57.714314 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.714287 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jdnl7\" (UniqueName: \"kubernetes.io/projected/818ac7a0-c51d-4a7a-9239-5447bf43477b-kube-api-access-jdnl7\") pod \"818ac7a0-c51d-4a7a-9239-5447bf43477b\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " Jul 24 16:20:57.714503 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.714356 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818ac7a0-c51d-4a7a-9239-5447bf43477b-plugins-config-volume\") pod \"818ac7a0-c51d-4a7a-9239-5447bf43477b\" (UID: \"818ac7a0-c51d-4a7a-9239-5447bf43477b\") " Jul 24 16:20:57.714755 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.714734 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818ac7a0-c51d-4a7a-9239-5447bf43477b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "818ac7a0-c51d-4a7a-9239-5447bf43477b" (UID: "818ac7a0-c51d-4a7a-9239-5447bf43477b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:20:57.716477 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.716453 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818ac7a0-c51d-4a7a-9239-5447bf43477b-kube-api-access-jdnl7" (OuterVolumeSpecName: "kube-api-access-jdnl7") pod "818ac7a0-c51d-4a7a-9239-5447bf43477b" (UID: "818ac7a0-c51d-4a7a-9239-5447bf43477b"). InnerVolumeSpecName "kube-api-access-jdnl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:20:57.815662 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.815626 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdnl7\" (UniqueName: \"kubernetes.io/projected/818ac7a0-c51d-4a7a-9239-5447bf43477b-kube-api-access-jdnl7\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:57.815662 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.815662 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/818ac7a0-c51d-4a7a-9239-5447bf43477b-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:20:57.942343 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.942312 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj"] Jul 24 16:20:57.946009 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:57.945984 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e8e158-547ffc854d-qc7dj"] Jul 24 16:20:59.043821 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:20:59.043790 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="818ac7a0-c51d-4a7a-9239-5447bf43477b" path="/var/lib/kubelet/pods/818ac7a0-c51d-4a7a-9239-5447bf43477b/volumes" Jul 24 16:21:08.861633 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:08.861595 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq"] Jul 24 16:21:08.862250 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:08.861888 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" containerID="cri-o://6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5" gracePeriod=30 Jul 24 16:21:09.704084 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.704005 2543 patch_prober.go:28] interesting pod/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 16:21:09.704084 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.704062 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 16:21:09.796397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796334 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c"] Jul 24 16:21:09.796916 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796893 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" Jul 24 16:21:09.796916 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796913 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" Jul 24 16:21:09.796916 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796922 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" Jul 24 16:21:09.797149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796927 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" Jul 24 16:21:09.797149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796963 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="818ac7a0-c51d-4a7a-9239-5447bf43477b" containerName="payload-pre-processing" Jul 24 16:21:09.797149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.796971 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="818ac7a0-c51d-4a7a-9239-5447bf43477b" containerName="payload-pre-processing" Jul 24 16:21:09.797149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.797049 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" Jul 24 16:21:09.797149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.797058 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b0c4ae9-1af1-4c66-ab1a-aacec7fb3ace" containerName="istio-proxy" Jul 24 16:21:09.797149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.797068 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="818ac7a0-c51d-4a7a-9239-5447bf43477b" containerName="payload-pre-processing" Jul 24 16:21:09.800224 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.800206 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.802706 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.802691 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-ed0643-openshift-default-dockercfg-tjznc\"" Jul 24 16:21:09.813859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.813836 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c"] Jul 24 16:21:09.941227 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941193 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-credential-socket\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941227 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941229 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-envoy\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941254 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jt66\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-kube-api-access-8jt66\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941272 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-token\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941306 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-certs\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941323 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-socket\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941364 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6449f-0f44-4255-9a43-07c7fb5d9224-istiod-ca-cert\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941393 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-podinfo\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:09.941689 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:09.941434 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-data\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042418 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042383 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6449f-0f44-4255-9a43-07c7fb5d9224-istiod-ca-cert\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042593 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042428 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-podinfo\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042593 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042556 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-data\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042698 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042681 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-credential-socket\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042743 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042728 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-envoy\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042796 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042764 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jt66\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-kube-api-access-8jt66\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042851 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042793 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-token\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042851 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042823 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-certs\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.042968 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.042853 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-socket\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.043074 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.043054 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-credential-socket\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.043157 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.043135 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-certs\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.043217 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.043186 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6449f-0f44-4255-9a43-07c7fb5d9224-istiod-ca-cert\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.043283 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.043266 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-socket\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.043514 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.043496 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-data\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.045093 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.045066 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-envoy\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.045212 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.045169 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-podinfo\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.052055 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.052027 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-token\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.052177 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.052161 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jt66\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-kube-api-access-8jt66\") pod \"e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.111113 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.111082 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:10.252103 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.252078 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c"] Jul 24 16:21:10.253783 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:21:10.253760 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed6449f_0f44_4255_9a43_07c7fb5d9224.slice/crio-d548833f921cfd23b488c29298a664aa13f8ccf99a6710ee7994a5263e2e88b9 WatchSource:0}: Error finding container d548833f921cfd23b488c29298a664aa13f8ccf99a6710ee7994a5263e2e88b9: Status 404 returned error can't find the container with id d548833f921cfd23b488c29298a664aa13f8ccf99a6710ee7994a5263e2e88b9 Jul 24 16:21:10.255875 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.255840 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:10.255964 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.255902 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:10.255964 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.255931 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:10.690006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.689919 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" event={"ID":"bed6449f-0f44-4255-9a43-07c7fb5d9224","Type":"ContainerStarted","Data":"6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc"} Jul 24 16:21:10.690006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.689977 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" event={"ID":"bed6449f-0f44-4255-9a43-07c7fb5d9224","Type":"ContainerStarted","Data":"d548833f921cfd23b488c29298a664aa13f8ccf99a6710ee7994a5263e2e88b9"} Jul 24 16:21:10.712807 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:10.712760 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" podStartSLOduration=1.712744069 podStartE2EDuration="1.712744069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:10.710746122 +0000 UTC m=+2402.242333754" watchObservedRunningTime="2026-07-24 16:21:10.712744069 +0000 UTC m=+2402.244331681" Jul 24 16:21:11.111531 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:11.111499 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:11.116617 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:11.116586 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:11.695590 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:11.695541 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:11.696593 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:11.696574 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:14.117501 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.117463 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:21:14.286236 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286197 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9khz6\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-kube-api-access-9khz6\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286421 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286313 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-socket\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286421 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286369 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-token\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286421 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286403 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-podinfo\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286798 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286432 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-certs\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286798 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286463 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-envoy\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286798 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286522 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-data\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286798 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286605 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5284567-ea1d-4658-b6a0-f13398624f1b-istiod-ca-cert\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286798 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286649 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-credential-socket\") pod \"d5284567-ea1d-4658-b6a0-f13398624f1b\" (UID: \"d5284567-ea1d-4658-b6a0-f13398624f1b\") " Jul 24 16:21:14.286798 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286678 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:14.287056 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.286991 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.287494 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.287473 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:14.288491 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.288431 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5284567-ea1d-4658-b6a0-f13398624f1b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:21:14.288753 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.288684 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:14.288753 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.288705 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-data" (OuterVolumeSpecName: "istio-data") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:14.289347 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.289319 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:21:14.289468 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.289357 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-kube-api-access-9khz6" (OuterVolumeSpecName: "kube-api-access-9khz6") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "kube-api-access-9khz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:21:14.289729 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.289697 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-token" (OuterVolumeSpecName: "istio-token") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:21:14.290635 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.290611 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d5284567-ea1d-4658-b6a0-f13398624f1b" (UID: "d5284567-ea1d-4658-b6a0-f13398624f1b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:21:14.388389 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388356 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388389 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388380 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388389 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388389 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388656 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388401 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388656 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388414 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5284567-ea1d-4658-b6a0-f13398624f1b-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388656 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388426 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5284567-ea1d-4658-b6a0-f13398624f1b-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388656 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388435 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9khz6\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-kube-api-access-9khz6\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.388656 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.388443 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5284567-ea1d-4658-b6a0-f13398624f1b-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:21:14.715965 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.715885 2543 generic.go:356] "Generic (PLEG): container finished" podID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerID="6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5" exitCode=0 Jul 24 16:21:14.715965 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.715924 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" event={"ID":"d5284567-ea1d-4658-b6a0-f13398624f1b","Type":"ContainerDied","Data":"6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5"} Jul 24 16:21:14.716161 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.715951 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" Jul 24 16:21:14.716161 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.716012 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq" event={"ID":"d5284567-ea1d-4658-b6a0-f13398624f1b","Type":"ContainerDied","Data":"4d97df42538f796da2a11122d751887ca88b1499a071a0ef524131dc29a22630"} Jul 24 16:21:14.716161 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.716048 2543 scope.go:117] "RemoveContainer" containerID="6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5" Jul 24 16:21:14.726776 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.726743 2543 scope.go:117] "RemoveContainer" containerID="c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117" Jul 24 16:21:14.735438 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.735420 2543 scope.go:117] "RemoveContainer" containerID="6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5" Jul 24 16:21:14.735760 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:21:14.735742 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5\": container with ID starting with 6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5 not found: ID does not exist" containerID="6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5" Jul 24 16:21:14.735813 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.735767 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5"} err="failed to get container status \"6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5\": rpc error: code = NotFound desc = could not find container \"6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5\": container with ID starting with 6d465160dccaf78ef2c154cf1db505b10d4210585f363ce1a02fff4032494fe5 not found: ID does not exist" Jul 24 16:21:14.735813 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.735784 2543 scope.go:117] "RemoveContainer" containerID="c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117" Jul 24 16:21:14.736012 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:21:14.735992 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117\": container with ID starting with c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117 not found: ID does not exist" containerID="c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117" Jul 24 16:21:14.736072 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.736022 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117"} err="failed to get container status \"c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117\": rpc error: code = NotFound desc = could not find container \"c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117\": container with ID starting with c2b11158cd3d398959f318001ce2891e7ad615b49110c1686f68382538af1117 not found: ID does not exist" Jul 24 16:21:14.742186 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.742163 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq"] Jul 24 16:21:14.746226 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:14.746203 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e8e158-openshift-default-69d5bf47b6-vl6jq"] Jul 24 16:21:15.045078 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:15.045047 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" path="/var/lib/kubelet/pods/d5284567-ea1d-4658-b6a0-f13398624f1b/volumes" Jul 24 16:21:17.883477 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.883442 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2"] Jul 24 16:21:17.884006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.883924 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" Jul 24 16:21:17.884006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.883941 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" Jul 24 16:21:17.884006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.883970 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" Jul 24 16:21:17.884006 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.883975 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" Jul 24 16:21:17.884247 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.884046 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" Jul 24 16:21:17.884247 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.884054 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5284567-ea1d-4658-b6a0-f13398624f1b" containerName="istio-proxy" Jul 24 16:21:17.887578 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.887546 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:17.890277 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.890260 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5aebfb-openshift-default-dockercfg-rfdds\"" Jul 24 16:21:17.899767 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:17.899740 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2"] Jul 24 16:21:18.034154 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034098 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954edbe4-2596-4121-8f65-0e386f75619a-istio-podinfo\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034367 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034165 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2f7w\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-kube-api-access-l2f7w\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034367 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034246 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-envoy\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034367 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034279 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-socket\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034367 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034307 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-istio-token\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034367 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034350 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954edbe4-2596-4121-8f65-0e386f75619a-istiod-ca-cert\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034695 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034377 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-certs\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034695 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034399 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-credential-socket\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.034695 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.034420 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-data\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135553 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135450 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954edbe4-2596-4121-8f65-0e386f75619a-istiod-ca-cert\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135553 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135511 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-certs\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135553 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135536 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-credential-socket\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135895 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135579 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-data\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135895 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135694 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954edbe4-2596-4121-8f65-0e386f75619a-istio-podinfo\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135895 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135733 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2f7w\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-kube-api-access-l2f7w\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135895 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135834 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-envoy\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.135895 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135880 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-socket\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.136165 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.135912 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-istio-token\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.137094 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.137060 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954edbe4-2596-4121-8f65-0e386f75619a-istiod-ca-cert\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.137227 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.137203 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-credential-socket\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.137299 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.137258 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-certs\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.137645 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.137608 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-socket\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.137785 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.137751 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-data\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.139814 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.139787 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-envoy\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.140484 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.140460 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954edbe4-2596-4121-8f65-0e386f75619a-istio-podinfo\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.145869 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.145844 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-istio-token\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.146439 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.146418 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2f7w\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-kube-api-access-l2f7w\") pod \"e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.198445 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.198411 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:18.396457 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.396434 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2"] Jul 24 16:21:18.398610 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:21:18.398583 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954edbe4_2596_4121_8f65_0e386f75619a.slice/crio-668c390ae4d9fee3295c7f59cd7da75b45c895dc28630c794d3cc0b68ea78883 WatchSource:0}: Error finding container 668c390ae4d9fee3295c7f59cd7da75b45c895dc28630c794d3cc0b68ea78883: Status 404 returned error can't find the container with id 668c390ae4d9fee3295c7f59cd7da75b45c895dc28630c794d3cc0b68ea78883 Jul 24 16:21:18.400930 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.400901 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:18.401011 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.400962 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:18.401011 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.400993 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:18.745850 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.745809 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" event={"ID":"954edbe4-2596-4121-8f65-0e386f75619a","Type":"ContainerStarted","Data":"deb4ddd557dd4b8bb2bd82e22144a8495d6fe5bbb38d01a04e8219a580678164"} Jul 24 16:21:18.746044 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.745863 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" event={"ID":"954edbe4-2596-4121-8f65-0e386f75619a","Type":"ContainerStarted","Data":"668c390ae4d9fee3295c7f59cd7da75b45c895dc28630c794d3cc0b68ea78883"} Jul 24 16:21:18.770624 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:18.770548 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podStartSLOduration=1.770524969 podStartE2EDuration="1.770524969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:18.768111352 +0000 UTC m=+2410.299698989" watchObservedRunningTime="2026-07-24 16:21:18.770524969 +0000 UTC m=+2410.302112583" Jul 24 16:21:19.199047 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:19.198965 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:19.204946 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:19.204920 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:19.751265 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:19.751234 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:19.752164 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:19.752145 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:22.101055 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.101018 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9"] Jul 24 16:21:22.104499 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.104472 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.108112 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.108063 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-5aebfb\"" Jul 24 16:21:22.108296 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.108205 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-5aebfb-dockercfg-sg45b\"" Jul 24 16:21:22.148038 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.148009 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9"] Jul 24 16:21:22.179749 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.179710 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdxb7\" (UniqueName: \"kubernetes.io/projected/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-kube-api-access-zdxb7\") pod \"payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.179922 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.179896 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.280545 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.280504 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.280545 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.280553 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdxb7\" (UniqueName: \"kubernetes.io/projected/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-kube-api-access-zdxb7\") pod \"payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.281205 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.281182 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.298257 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.298234 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdxb7\" (UniqueName: \"kubernetes.io/projected/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-kube-api-access-zdxb7\") pod \"payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.419620 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.419505 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.578312 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.578288 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9"] Jul 24 16:21:22.579948 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:21:22.579916 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76f4caf0_f7d0_4222_8b26_d6d2f90a9415.slice/crio-1759f99f06fa5dc0a9c96580b22e5498ea40c80f3f1475dc2b5473a99dadb230 WatchSource:0}: Error finding container 1759f99f06fa5dc0a9c96580b22e5498ea40c80f3f1475dc2b5473a99dadb230: Status 404 returned error can't find the container with id 1759f99f06fa5dc0a9c96580b22e5498ea40c80f3f1475dc2b5473a99dadb230 Jul 24 16:21:22.771140 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.771071 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" event={"ID":"76f4caf0-f7d0-4222-8b26-d6d2f90a9415","Type":"ContainerStarted","Data":"31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a"} Jul 24 16:21:22.771140 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.771144 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:22.771376 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.771165 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" event={"ID":"76f4caf0-f7d0-4222-8b26-d6d2f90a9415","Type":"ContainerStarted","Data":"1759f99f06fa5dc0a9c96580b22e5498ea40c80f3f1475dc2b5473a99dadb230"} Jul 24 16:21:22.789877 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:22.789826 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" podStartSLOduration=0.789809699 podStartE2EDuration="789.809699ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:21:22.787371629 +0000 UTC m=+2414.318959260" watchObservedRunningTime="2026-07-24 16:21:22.789809699 +0000 UTC m=+2414.321397312" Jul 24 16:21:24.783973 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:24.783941 2543 generic.go:356] "Generic (PLEG): container finished" podID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerID="6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc" exitCode=0 Jul 24 16:21:24.784386 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:24.784009 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" event={"ID":"bed6449f-0f44-4255-9a43-07c7fb5d9224","Type":"ContainerDied","Data":"6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc"} Jul 24 16:21:24.784433 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:24.784425 2543 scope.go:117] "RemoveContainer" containerID="6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc" Jul 24 16:21:24.785485 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:24.785459 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:24.785536 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:24.785524 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:24.785587 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:24.785575 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:25.791969 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:25.791926 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" event={"ID":"bed6449f-0f44-4255-9a43-07c7fb5d9224","Type":"ContainerStarted","Data":"8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb"} Jul 24 16:21:26.112650 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:26.112528 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:26.112846 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:26.112691 2543 patch_prober.go:28] interesting pod/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 24 16:21:26.112846 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:26.112739 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 24 16:21:26.695805 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:26.695774 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:27.112461 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:27.112423 2543 patch_prober.go:28] interesting pod/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 24 16:21:27.113057 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:27.113025 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 24 16:21:28.112226 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:28.112188 2543 patch_prober.go:28] interesting pod/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 24 16:21:28.112505 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:28.112474 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 24 16:21:29.113048 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:29.112953 2543 patch_prober.go:28] interesting pod/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 24 16:21:29.113655 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:29.113620 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 24 16:21:30.116445 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:30.116417 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:30.117452 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:30.117429 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:21:33.779821 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.779784 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:21:33.849999 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.849961 2543 generic.go:356] "Generic (PLEG): container finished" podID="954edbe4-2596-4121-8f65-0e386f75619a" containerID="deb4ddd557dd4b8bb2bd82e22144a8495d6fe5bbb38d01a04e8219a580678164" exitCode=0 Jul 24 16:21:33.850162 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.850031 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" event={"ID":"954edbe4-2596-4121-8f65-0e386f75619a","Type":"ContainerDied","Data":"deb4ddd557dd4b8bb2bd82e22144a8495d6fe5bbb38d01a04e8219a580678164"} Jul 24 16:21:33.850456 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.850442 2543 scope.go:117] "RemoveContainer" containerID="deb4ddd557dd4b8bb2bd82e22144a8495d6fe5bbb38d01a04e8219a580678164" Jul 24 16:21:33.851535 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.851503 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:33.851631 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.851591 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:33.851679 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:33.851642 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:21:34.751546 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:34.751513 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:34.857925 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:34.857892 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" event={"ID":"954edbe4-2596-4121-8f65-0e386f75619a","Type":"ContainerStarted","Data":"e10c98831099c100762916fbc5e2bcc00f6499a59fcbc615266aed2c325f5345"} Jul 24 16:21:35.199634 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:35.199508 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:35.200794 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:35.200770 2543 patch_prober.go:28] interesting pod/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 16:21:35.200910 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:35.200821 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 16:21:36.199035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:36.199002 2543 patch_prober.go:28] interesting pod/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 16:21:36.199408 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:36.199058 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 16:21:37.199728 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:37.199694 2543 patch_prober.go:28] interesting pod/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 16:21:37.200181 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:37.199780 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 16:21:38.199280 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:38.199250 2543 patch_prober.go:28] interesting pod/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Jul 24 16:21:38.199453 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:38.199310 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Jul 24 16:21:39.203397 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:39.203372 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:39.203795 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:39.203645 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:21:39.204395 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:21:39.204381 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:28:09.288350 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.288300 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w"] Jul 24 16:28:09.292985 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.292964 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.295575 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.295537 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-2c5dfe-openshift-default-dockercfg-m7glg\"" Jul 24 16:28:09.302995 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.302974 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w"] Jul 24 16:28:09.380805 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.380749 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393fa908-c4be-4b8f-baec-cdb9d9660a77-istiod-ca-cert\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.380805 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.380811 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-envoy\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381087 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.380906 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-socket\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381087 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.380943 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njpfh\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-kube-api-access-njpfh\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381087 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.381040 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-podinfo\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381264 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.381084 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-certs\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381264 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.381196 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-credential-socket\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381264 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.381240 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-data\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.381427 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.381312 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-token\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.482628 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482594 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-credential-socket\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.482628 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482635 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-data\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.482889 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482665 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-token\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.482889 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482734 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393fa908-c4be-4b8f-baec-cdb9d9660a77-istiod-ca-cert\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.482889 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482770 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-envoy\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.482889 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482802 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-socket\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483099 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.482935 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njpfh\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-kube-api-access-njpfh\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483099 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483008 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-podinfo\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483099 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483057 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-certs\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483099 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483078 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-credential-socket\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483285 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483102 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-data\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483419 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483399 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-socket\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483478 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483458 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-certs\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.483530 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.483465 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393fa908-c4be-4b8f-baec-cdb9d9660a77-istiod-ca-cert\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.485433 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.485388 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-envoy\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.485772 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.485753 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-podinfo\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.490981 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.490953 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njpfh\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-kube-api-access-njpfh\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.491110 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.491015 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-token\") pod \"e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.604938 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.604838 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:09.751497 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.751453 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w"] Jul 24 16:28:09.755030 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:28:09.755001 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod393fa908_c4be_4b8f_baec_cdb9d9660a77.slice/crio-9826320aa5de5139c7243b2700e63a2433d72e318c667597ebe9f53e14762515 WatchSource:0}: Error finding container 9826320aa5de5139c7243b2700e63a2433d72e318c667597ebe9f53e14762515: Status 404 returned error can't find the container with id 9826320aa5de5139c7243b2700e63a2433d72e318c667597ebe9f53e14762515 Jul 24 16:28:09.757528 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.757496 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:28:09.757671 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.757595 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:28:09.757671 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.757647 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:28:09.986078 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.986042 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" event={"ID":"393fa908-c4be-4b8f-baec-cdb9d9660a77","Type":"ContainerStarted","Data":"5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac"} Jul 24 16:28:09.986298 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:09.986103 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" event={"ID":"393fa908-c4be-4b8f-baec-cdb9d9660a77","Type":"ContainerStarted","Data":"9826320aa5de5139c7243b2700e63a2433d72e318c667597ebe9f53e14762515"} Jul 24 16:28:10.005422 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:10.005365 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podStartSLOduration=1.005347792 podStartE2EDuration="1.005347792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:28:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:28:10.004086024 +0000 UTC m=+2821.535673654" watchObservedRunningTime="2026-07-24 16:28:10.005347792 +0000 UTC m=+2821.536935404" Jul 24 16:28:10.605811 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:10.605772 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:10.611476 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:10.611449 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:10.992199 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:10.992172 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:10.993034 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:10.993017 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:12.788019 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.787979 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h"] Jul 24 16:28:12.792354 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.792308 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:12.794951 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.794929 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-2c5dfe\"" Jul 24 16:28:12.795080 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.795026 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-2c5dfe-dockercfg-fzllc\"" Jul 24 16:28:12.802816 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.802791 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h"] Jul 24 16:28:12.922157 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.922106 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvkz2\" (UniqueName: \"kubernetes.io/projected/a135a852-932c-4d47-a160-1c6c2f9d2536-kube-api-access-tvkz2\") pod \"payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:12.922320 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:12.922164 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a135a852-932c-4d47-a160-1c6c2f9d2536-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:13.023075 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:13.023041 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvkz2\" (UniqueName: \"kubernetes.io/projected/a135a852-932c-4d47-a160-1c6c2f9d2536-kube-api-access-tvkz2\") pod \"payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:13.023242 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:13.023080 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a135a852-932c-4d47-a160-1c6c2f9d2536-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:13.023860 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:13.023826 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a135a852-932c-4d47-a160-1c6c2f9d2536-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:13.031513 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:13.031488 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvkz2\" (UniqueName: \"kubernetes.io/projected/a135a852-932c-4d47-a160-1c6c2f9d2536-kube-api-access-tvkz2\") pod \"payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:13.104224 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:13.104158 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:13.246783 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:13.246758 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h"] Jul 24 16:28:13.248322 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:28:13.248284 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda135a852_932c_4d47_a160_1c6c2f9d2536.slice/crio-3972070e21fd2f57d9f7b2a4a70fa19344471c0e423cc76612d3b95ac01ba50e WatchSource:0}: Error finding container 3972070e21fd2f57d9f7b2a4a70fa19344471c0e423cc76612d3b95ac01ba50e: Status 404 returned error can't find the container with id 3972070e21fd2f57d9f7b2a4a70fa19344471c0e423cc76612d3b95ac01ba50e Jul 24 16:28:14.012727 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:14.012687 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" event={"ID":"a135a852-932c-4d47-a160-1c6c2f9d2536","Type":"ContainerStarted","Data":"89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45"} Jul 24 16:28:14.013192 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:14.012751 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" event={"ID":"a135a852-932c-4d47-a160-1c6c2f9d2536","Type":"ContainerStarted","Data":"3972070e21fd2f57d9f7b2a4a70fa19344471c0e423cc76612d3b95ac01ba50e"} Jul 24 16:28:14.013192 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:14.012838 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:14.032649 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:14.032590 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" podStartSLOduration=2.032570686 podStartE2EDuration="2.032570686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:28:14.029945785 +0000 UTC m=+2825.561533401" watchObservedRunningTime="2026-07-24 16:28:14.032570686 +0000 UTC m=+2825.564158294" Jul 24 16:28:25.021145 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:25.021105 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:28:27.034716 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:27.034675 2543 patch_prober.go:28] interesting pod/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 16:28:27.035096 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:27.034746 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 16:28:29.098834 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:29.098802 2543 generic.go:356] "Generic (PLEG): container finished" podID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerID="5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac" exitCode=0 Jul 24 16:28:29.099256 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:29.098855 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" event={"ID":"393fa908-c4be-4b8f-baec-cdb9d9660a77","Type":"ContainerDied","Data":"5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac"} Jul 24 16:28:29.099256 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:29.099223 2543 scope.go:117] "RemoveContainer" containerID="5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac" Jul 24 16:28:29.100339 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:29.100309 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:28:29.100424 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:29.100393 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:28:29.100462 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:29.100424 2543 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 16:28:30.107342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:30.106572 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" event={"ID":"393fa908-c4be-4b8f-baec-cdb9d9660a77","Type":"ContainerStarted","Data":"22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6"} Jul 24 16:28:30.606391 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:30.606346 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:30.606643 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:30.606490 2543 patch_prober.go:28] interesting pod/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 16:28:30.606643 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:30.606550 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 16:28:31.606127 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:31.606084 2543 patch_prober.go:28] interesting pod/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 16:28:31.606654 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:31.606162 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 16:28:32.605774 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:32.605732 2543 patch_prober.go:28] interesting pod/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 16:28:32.605985 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:32.605796 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 16:28:33.606761 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:33.606278 2543 patch_prober.go:28] interesting pod/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Jul 24 16:28:33.606761 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:33.606341 2543 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Jul 24 16:28:34.618782 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:34.618749 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:34.619233 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:34.619180 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:34.619935 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:34.619915 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:28:55.020966 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:28:55.020917 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lj4t_9d58e57f-6052-4e74-9cf5-6114eb69024f/payload-processing/0.log" Jul 24 16:29:54.279360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:54.279328 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h_a135a852-932c-4d47-a160-1c6c2f9d2536/payload-processing/0.log" Jul 24 16:29:54.404862 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:54.404827 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lj4t_9d58e57f-6052-4e74-9cf5-6114eb69024f/payload-processing/0.log" Jul 24 16:29:58.638831 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.638799 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk"] Jul 24 16:29:58.642387 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.642372 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.646052 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.646028 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-4c4c5c-dockercfg-j5z9m\"" Jul 24 16:29:58.646188 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.646069 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-4c4c5c\"" Jul 24 16:29:58.654375 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.654350 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk"] Jul 24 16:29:58.729709 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.729670 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62d562-0eca-49c5-a05f-a75c3226f3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.729884 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.729743 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9bs5\" (UniqueName: \"kubernetes.io/projected/2c62d562-0eca-49c5-a05f-a75c3226f3ef-kube-api-access-p9bs5\") pod \"payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.830715 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.830681 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62d562-0eca-49c5-a05f-a75c3226f3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.830906 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.830745 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9bs5\" (UniqueName: \"kubernetes.io/projected/2c62d562-0eca-49c5-a05f-a75c3226f3ef-kube-api-access-p9bs5\") pod \"payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.831375 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.831356 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62d562-0eca-49c5-a05f-a75c3226f3ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.838577 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.838545 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9bs5\" (UniqueName: \"kubernetes.io/projected/2c62d562-0eca-49c5-a05f-a75c3226f3ef-kube-api-access-p9bs5\") pod \"payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:58.954449 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:58.954366 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:59.097416 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:59.097387 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk"] Jul 24 16:29:59.099314 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:29:59.099286 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c62d562_0eca_49c5_a05f_a75c3226f3ef.slice/crio-552a9d71a6a2710dbe35e4378d6471be1b62e2dad7a6f569a7ac4f31045b4247 WatchSource:0}: Error finding container 552a9d71a6a2710dbe35e4378d6471be1b62e2dad7a6f569a7ac4f31045b4247: Status 404 returned error can't find the container with id 552a9d71a6a2710dbe35e4378d6471be1b62e2dad7a6f569a7ac4f31045b4247 Jul 24 16:29:59.619477 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:59.619428 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" event={"ID":"2c62d562-0eca-49c5-a05f-a75c3226f3ef","Type":"ContainerStarted","Data":"27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3"} Jul 24 16:29:59.619685 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:59.619498 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" event={"ID":"2c62d562-0eca-49c5-a05f-a75c3226f3ef","Type":"ContainerStarted","Data":"552a9d71a6a2710dbe35e4378d6471be1b62e2dad7a6f569a7ac4f31045b4247"} Jul 24 16:29:59.619685 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:59.619653 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:29:59.635278 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:29:59.635223 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" podStartSLOduration=1.635206318 podStartE2EDuration="1.635206318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:29:59.633189252 +0000 UTC m=+2931.164776903" watchObservedRunningTime="2026-07-24 16:29:59.635206318 +0000 UTC m=+2931.166793930" Jul 24 16:30:10.627632 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:10.627598 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:30:23.805199 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:23.805167 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk"] Jul 24 16:30:23.805828 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:23.805426 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" podUID="2c62d562-0eca-49c5-a05f-a75c3226f3ef" containerName="payload-pre-processing" containerID="cri-o://27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3" gracePeriod=30 Jul 24 16:30:24.070632 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.070546 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:30:24.094194 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.094162 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p9bs5\" (UniqueName: \"kubernetes.io/projected/2c62d562-0eca-49c5-a05f-a75c3226f3ef-kube-api-access-p9bs5\") pod \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " Jul 24 16:30:24.094331 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.094274 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62d562-0eca-49c5-a05f-a75c3226f3ef-plugins-config-volume\") pod \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\" (UID: \"2c62d562-0eca-49c5-a05f-a75c3226f3ef\") " Jul 24 16:30:24.095045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.094650 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c62d562-0eca-49c5-a05f-a75c3226f3ef-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2c62d562-0eca-49c5-a05f-a75c3226f3ef" (UID: "2c62d562-0eca-49c5-a05f-a75c3226f3ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:30:24.095205 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.095135 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c62d562-0eca-49c5-a05f-a75c3226f3ef-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:30:24.096601 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.096533 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c62d562-0eca-49c5-a05f-a75c3226f3ef-kube-api-access-p9bs5" (OuterVolumeSpecName: "kube-api-access-p9bs5") pod "2c62d562-0eca-49c5-a05f-a75c3226f3ef" (UID: "2c62d562-0eca-49c5-a05f-a75c3226f3ef"). InnerVolumeSpecName "kube-api-access-p9bs5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:30:24.196068 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.196039 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9bs5\" (UniqueName: \"kubernetes.io/projected/2c62d562-0eca-49c5-a05f-a75c3226f3ef-kube-api-access-p9bs5\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:30:24.774352 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.774318 2543 generic.go:356] "Generic (PLEG): container finished" podID="2c62d562-0eca-49c5-a05f-a75c3226f3ef" containerID="27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3" exitCode=0 Jul 24 16:30:24.774597 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.774361 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" Jul 24 16:30:24.774597 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.774404 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" event={"ID":"2c62d562-0eca-49c5-a05f-a75c3226f3ef","Type":"ContainerDied","Data":"27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3"} Jul 24 16:30:24.774597 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.774461 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk" event={"ID":"2c62d562-0eca-49c5-a05f-a75c3226f3ef","Type":"ContainerDied","Data":"552a9d71a6a2710dbe35e4378d6471be1b62e2dad7a6f569a7ac4f31045b4247"} Jul 24 16:30:24.774597 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.774490 2543 scope.go:117] "RemoveContainer" containerID="27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3" Jul 24 16:30:24.784871 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.784781 2543 scope.go:117] "RemoveContainer" containerID="27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3" Jul 24 16:30:24.787023 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:30:24.787000 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3\": container with ID starting with 27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3 not found: ID does not exist" containerID="27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3" Jul 24 16:30:24.787162 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.787028 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3"} err="failed to get container status \"27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3\": rpc error: code = NotFound desc = could not find container \"27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3\": container with ID starting with 27b1be30d066b63c39bbe37486c71e7f8446e8bebbf9d24fb3b5caf12f1139d3 not found: ID does not exist" Jul 24 16:30:24.795314 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.795291 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk"] Jul 24 16:30:24.798751 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:24.798730 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-4c4c5c-6d96d574f6-sghdk"] Jul 24 16:30:25.043912 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:25.043827 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2c62d562-0eca-49c5-a05f-a75c3226f3ef" path="/var/lib/kubelet/pods/2c62d562-0eca-49c5-a05f-a75c3226f3ef/volumes" Jul 24 16:30:53.913316 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:53.913278 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h"] Jul 24 16:30:53.914003 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:53.913526 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" podUID="a135a852-932c-4d47-a160-1c6c2f9d2536" containerName="payload-processing" containerID="cri-o://89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45" gracePeriod=30 Jul 24 16:30:54.202871 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.202843 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:30:54.214463 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.214435 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvkz2\" (UniqueName: \"kubernetes.io/projected/a135a852-932c-4d47-a160-1c6c2f9d2536-kube-api-access-tvkz2\") pod \"a135a852-932c-4d47-a160-1c6c2f9d2536\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " Jul 24 16:30:54.214656 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.214480 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a135a852-932c-4d47-a160-1c6c2f9d2536-plugins-config-volume\") pod \"a135a852-932c-4d47-a160-1c6c2f9d2536\" (UID: \"a135a852-932c-4d47-a160-1c6c2f9d2536\") " Jul 24 16:30:54.214913 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.214884 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a135a852-932c-4d47-a160-1c6c2f9d2536-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a135a852-932c-4d47-a160-1c6c2f9d2536" (UID: "a135a852-932c-4d47-a160-1c6c2f9d2536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:30:54.216778 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.216755 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a135a852-932c-4d47-a160-1c6c2f9d2536-kube-api-access-tvkz2" (OuterVolumeSpecName: "kube-api-access-tvkz2") pod "a135a852-932c-4d47-a160-1c6c2f9d2536" (UID: "a135a852-932c-4d47-a160-1c6c2f9d2536"). InnerVolumeSpecName "kube-api-access-tvkz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:30:54.315963 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.315928 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvkz2\" (UniqueName: \"kubernetes.io/projected/a135a852-932c-4d47-a160-1c6c2f9d2536-kube-api-access-tvkz2\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:30:54.315963 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.315955 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a135a852-932c-4d47-a160-1c6c2f9d2536-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:30:54.951249 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.951212 2543 generic.go:356] "Generic (PLEG): container finished" podID="a135a852-932c-4d47-a160-1c6c2f9d2536" containerID="89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45" exitCode=0 Jul 24 16:30:54.951793 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.951261 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" Jul 24 16:30:54.951793 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.951269 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" event={"ID":"a135a852-932c-4d47-a160-1c6c2f9d2536","Type":"ContainerDied","Data":"89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45"} Jul 24 16:30:54.951793 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.951340 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h" event={"ID":"a135a852-932c-4d47-a160-1c6c2f9d2536","Type":"ContainerDied","Data":"3972070e21fd2f57d9f7b2a4a70fa19344471c0e423cc76612d3b95ac01ba50e"} Jul 24 16:30:54.951793 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.951376 2543 scope.go:117] "RemoveContainer" containerID="89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45" Jul 24 16:30:54.961045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.961028 2543 scope.go:117] "RemoveContainer" containerID="89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45" Jul 24 16:30:54.961331 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:30:54.961302 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45\": container with ID starting with 89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45 not found: ID does not exist" containerID="89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45" Jul 24 16:30:54.961384 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.961346 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45"} err="failed to get container status \"89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45\": rpc error: code = NotFound desc = could not find container \"89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45\": container with ID starting with 89a0aab5d14a075aa2e993455e31ae215844afcec3b06b732b061697921cfb45 not found: ID does not exist" Jul 24 16:30:54.973292 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.973268 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h"] Jul 24 16:30:54.976919 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:54.976898 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-2c5dfe-78fc4cfbfb-v5n4h"] Jul 24 16:30:55.045054 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:30:55.045022 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a135a852-932c-4d47-a160-1c6c2f9d2536" path="/var/lib/kubelet/pods/a135a852-932c-4d47-a160-1c6c2f9d2536/volumes" Jul 24 16:31:46.517284 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:46.517242 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w"] Jul 24 16:31:46.517920 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:46.517494 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" containerID="cri-o://22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6" gracePeriod=30 Jul 24 16:31:49.621580 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:49.621530 2543 patch_prober.go:28] interesting pod/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 16:31:49.621968 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:49.621616 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 16:31:51.789475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.789416 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:31:51.905952 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.905917 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-certs\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.905952 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.905952 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-envoy\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906208 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.905987 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-credential-socket\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906208 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906023 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-token\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906208 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906052 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-data\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906208 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906095 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393fa908-c4be-4b8f-baec-cdb9d9660a77-istiod-ca-cert\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906208 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906140 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-podinfo\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906206 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:31:51.906460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906296 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:31:51.906586 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906472 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/393fa908-c4be-4b8f-baec-cdb9d9660a77-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:31:51.906667 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906647 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njpfh\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-kube-api-access-njpfh\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.906767 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.906750 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-socket\") pod \"393fa908-c4be-4b8f-baec-cdb9d9660a77\" (UID: \"393fa908-c4be-4b8f-baec-cdb9d9660a77\") " Jul 24 16:31:51.907055 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907029 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:31:51.907055 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907037 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-data" (OuterVolumeSpecName: "istio-data") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:31:51.907426 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907404 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:51.907625 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907601 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:51.907760 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907626 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:51.907760 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907641 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:51.907760 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.907653 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/393fa908-c4be-4b8f-baec-cdb9d9660a77-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:51.908376 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.908346 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:31:51.908512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.908494 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-token" (OuterVolumeSpecName: "istio-token") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:31:51.909016 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.909002 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:31:51.909206 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:51.909192 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-kube-api-access-njpfh" (OuterVolumeSpecName: "kube-api-access-njpfh") pod "393fa908-c4be-4b8f-baec-cdb9d9660a77" (UID: "393fa908-c4be-4b8f-baec-cdb9d9660a77"). InnerVolumeSpecName "kube-api-access-njpfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:31:52.009198 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.009160 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:52.009198 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.009191 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:52.009198 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.009203 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/393fa908-c4be-4b8f-baec-cdb9d9660a77-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:52.009438 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.009213 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njpfh\" (UniqueName: \"kubernetes.io/projected/393fa908-c4be-4b8f-baec-cdb9d9660a77-kube-api-access-njpfh\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:31:52.264313 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.264274 2543 generic.go:356] "Generic (PLEG): container finished" podID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerID="22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6" exitCode=0 Jul 24 16:31:52.264499 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.264326 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" Jul 24 16:31:52.264499 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.264359 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" event={"ID":"393fa908-c4be-4b8f-baec-cdb9d9660a77","Type":"ContainerDied","Data":"22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6"} Jul 24 16:31:52.264499 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.264417 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w" event={"ID":"393fa908-c4be-4b8f-baec-cdb9d9660a77","Type":"ContainerDied","Data":"9826320aa5de5139c7243b2700e63a2433d72e318c667597ebe9f53e14762515"} Jul 24 16:31:52.264499 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.264438 2543 scope.go:117] "RemoveContainer" containerID="22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6" Jul 24 16:31:52.274427 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.274409 2543 scope.go:117] "RemoveContainer" containerID="5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac" Jul 24 16:31:52.282748 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.282728 2543 scope.go:117] "RemoveContainer" containerID="22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6" Jul 24 16:31:52.282992 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:31:52.282974 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6\": container with ID starting with 22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6 not found: ID does not exist" containerID="22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6" Jul 24 16:31:52.283041 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.283002 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6"} err="failed to get container status \"22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6\": rpc error: code = NotFound desc = could not find container \"22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6\": container with ID starting with 22b471e5b94e703ef01d2523a6eb6f82023d35c8593cfc946eba8e6c93bf14d6 not found: ID does not exist" Jul 24 16:31:52.283041 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.283020 2543 scope.go:117] "RemoveContainer" containerID="5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac" Jul 24 16:31:52.283275 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:31:52.283258 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac\": container with ID starting with 5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac not found: ID does not exist" containerID="5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac" Jul 24 16:31:52.283315 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.283282 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac"} err="failed to get container status \"5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac\": rpc error: code = NotFound desc = could not find container \"5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac\": container with ID starting with 5774c98925916ef3b179ef6472a2019f6d2125d3c84c703d3916d34d777c8eac not found: ID does not exist" Jul 24 16:31:52.287159 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.287136 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w"] Jul 24 16:31:52.291256 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:52.291237 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-2c5dfe-openshift-default-d9db4b8f6-fwh6w"] Jul 24 16:31:53.044145 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:31:53.044111 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" path="/var/lib/kubelet/pods/393fa908-c4be-4b8f-baec-cdb9d9660a77/volumes" Jul 24 16:32:30.442688 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:30.442654 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c"] Jul 24 16:32:30.443293 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:30.442892 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" containerID="cri-o://8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb" gracePeriod=30 Jul 24 16:32:35.686881 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.686856 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:32:35.838477 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838447 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-data\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838527 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8jt66\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-kube-api-access-8jt66\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838551 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-socket\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838604 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6449f-0f44-4255-9a43-07c7fb5d9224-istiod-ca-cert\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838631 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-envoy\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838663 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-podinfo\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838703 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-credential-socket\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838775 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-certs\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.838804 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838802 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-token\") pod \"bed6449f-0f44-4255-9a43-07c7fb5d9224\" (UID: \"bed6449f-0f44-4255-9a43-07c7fb5d9224\") " Jul 24 16:32:35.839264 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.838999 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:35.839264 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839084 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:35.839264 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839098 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bed6449f-0f44-4255-9a43-07c7fb5d9224-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:32:35.839581 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839279 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:35.839581 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839329 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.839581 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839348 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.839581 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839362 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bed6449f-0f44-4255-9a43-07c7fb5d9224-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.839581 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.839394 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-data" (OuterVolumeSpecName: "istio-data") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:35.841062 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.841035 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:32:35.841062 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.841052 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:35.841230 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.841130 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-kube-api-access-8jt66" (OuterVolumeSpecName: "kube-api-access-8jt66") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "kube-api-access-8jt66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:35.841269 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.841246 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-token" (OuterVolumeSpecName: "istio-token") pod "bed6449f-0f44-4255-9a43-07c7fb5d9224" (UID: "bed6449f-0f44-4255-9a43-07c7fb5d9224"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:35.940670 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.940639 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.940670 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.940669 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.940670 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.940678 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.940670 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.940685 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jt66\" (UniqueName: \"kubernetes.io/projected/bed6449f-0f44-4255-9a43-07c7fb5d9224-kube-api-access-8jt66\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.940950 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.940694 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:35.940950 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:35.940703 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bed6449f-0f44-4255-9a43-07c7fb5d9224-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:36.520912 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.520881 2543 generic.go:356] "Generic (PLEG): container finished" podID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerID="8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb" exitCode=0 Jul 24 16:32:36.521086 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.520931 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" Jul 24 16:32:36.521086 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.520945 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" event={"ID":"bed6449f-0f44-4255-9a43-07c7fb5d9224","Type":"ContainerDied","Data":"8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb"} Jul 24 16:32:36.521086 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.520988 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c" event={"ID":"bed6449f-0f44-4255-9a43-07c7fb5d9224","Type":"ContainerDied","Data":"d548833f921cfd23b488c29298a664aa13f8ccf99a6710ee7994a5263e2e88b9"} Jul 24 16:32:36.521086 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.521010 2543 scope.go:117] "RemoveContainer" containerID="8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb" Jul 24 16:32:36.534287 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.534269 2543 scope.go:117] "RemoveContainer" containerID="6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc" Jul 24 16:32:36.542472 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.542453 2543 scope.go:117] "RemoveContainer" containerID="8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb" Jul 24 16:32:36.542770 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:32:36.542747 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb\": container with ID starting with 8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb not found: ID does not exist" containerID="8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb" Jul 24 16:32:36.542859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.542778 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb"} err="failed to get container status \"8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb\": rpc error: code = NotFound desc = could not find container \"8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb\": container with ID starting with 8e291b303166ab202b4a7989f320370b60167bbeda2caf1d65b5b2d9178d14eb not found: ID does not exist" Jul 24 16:32:36.542859 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.542793 2543 scope.go:117] "RemoveContainer" containerID="6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc" Jul 24 16:32:36.543049 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:32:36.543029 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc\": container with ID starting with 6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc not found: ID does not exist" containerID="6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc" Jul 24 16:32:36.543104 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.543057 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc"} err="failed to get container status \"6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc\": rpc error: code = NotFound desc = could not find container \"6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc\": container with ID starting with 6577eadb42caa3275f9f2f8a51af31cca2ab2abbfe16070f38586d8908c709bc not found: ID does not exist" Jul 24 16:32:36.546724 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.546703 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c"] Jul 24 16:32:36.549697 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:36.549675 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-ed0643-openshift-default-5c6bbfc94d-fql8c"] Jul 24 16:32:37.044210 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:37.044177 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" path="/var/lib/kubelet/pods/bed6449f-0f44-4255-9a43-07c7fb5d9224/volumes" Jul 24 16:32:41.432960 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.432926 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9"] Jul 24 16:32:41.433403 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.433157 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" podUID="76f4caf0-f7d0-4222-8b26-d6d2f90a9415" containerName="payload-processing" containerID="cri-o://31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a" gracePeriod=30 Jul 24 16:32:41.707016 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.706990 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:32:41.900239 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.900207 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdxb7\" (UniqueName: \"kubernetes.io/projected/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-kube-api-access-zdxb7\") pod \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " Jul 24 16:32:41.900436 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.900294 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-plugins-config-volume\") pod \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\" (UID: \"76f4caf0-f7d0-4222-8b26-d6d2f90a9415\") " Jul 24 16:32:41.900619 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.900595 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76f4caf0-f7d0-4222-8b26-d6d2f90a9415" (UID: "76f4caf0-f7d0-4222-8b26-d6d2f90a9415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:32:41.902420 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:41.902401 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-kube-api-access-zdxb7" (OuterVolumeSpecName: "kube-api-access-zdxb7") pod "76f4caf0-f7d0-4222-8b26-d6d2f90a9415" (UID: "76f4caf0-f7d0-4222-8b26-d6d2f90a9415"). InnerVolumeSpecName "kube-api-access-zdxb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:42.001768 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.001738 2543 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-plugins-config-volume\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:42.001768 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.001763 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdxb7\" (UniqueName: \"kubernetes.io/projected/76f4caf0-f7d0-4222-8b26-d6d2f90a9415-kube-api-access-zdxb7\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:42.554942 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.554909 2543 generic.go:356] "Generic (PLEG): container finished" podID="76f4caf0-f7d0-4222-8b26-d6d2f90a9415" containerID="31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a" exitCode=0 Jul 24 16:32:42.555383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.554954 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" Jul 24 16:32:42.555383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.554995 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" event={"ID":"76f4caf0-f7d0-4222-8b26-d6d2f90a9415","Type":"ContainerDied","Data":"31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a"} Jul 24 16:32:42.555383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.555046 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9" event={"ID":"76f4caf0-f7d0-4222-8b26-d6d2f90a9415","Type":"ContainerDied","Data":"1759f99f06fa5dc0a9c96580b22e5498ea40c80f3f1475dc2b5473a99dadb230"} Jul 24 16:32:42.555383 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.555068 2543 scope.go:117] "RemoveContainer" containerID="31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a" Jul 24 16:32:42.564460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.564442 2543 scope.go:117] "RemoveContainer" containerID="31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a" Jul 24 16:32:42.564763 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:32:42.564744 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a\": container with ID starting with 31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a not found: ID does not exist" containerID="31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a" Jul 24 16:32:42.564827 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.564773 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a"} err="failed to get container status \"31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a\": rpc error: code = NotFound desc = could not find container \"31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a\": container with ID starting with 31486d7e77810d54eaabc25119a5ba5469a0a2481f686a185d82bbd92da3721a not found: ID does not exist" Jul 24 16:32:42.576167 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.576130 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9"] Jul 24 16:32:42.581188 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:42.581165 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-5aebfb-698c45b98b-4djt9"] Jul 24 16:32:43.045294 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:43.045263 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76f4caf0-f7d0-4222-8b26-d6d2f90a9415" path="/var/lib/kubelet/pods/76f4caf0-f7d0-4222-8b26-d6d2f90a9415/volumes" Jul 24 16:32:52.454784 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:52.454748 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2"] Jul 24 16:32:52.455262 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:52.455036 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" containerID="cri-o://e10c98831099c100762916fbc5e2bcc00f6499a59fcbc615266aed2c325f5345" gracePeriod=30 Jul 24 16:32:54.205970 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:54.205930 2543 patch_prober.go:28] interesting pod/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 16:32:54.206421 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:54.206003 2543 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 16:32:57.638683 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.638653 2543 generic.go:356] "Generic (PLEG): container finished" podID="954edbe4-2596-4121-8f65-0e386f75619a" containerID="e10c98831099c100762916fbc5e2bcc00f6499a59fcbc615266aed2c325f5345" exitCode=0 Jul 24 16:32:57.639045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.638731 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" event={"ID":"954edbe4-2596-4121-8f65-0e386f75619a","Type":"ContainerDied","Data":"e10c98831099c100762916fbc5e2bcc00f6499a59fcbc615266aed2c325f5345"} Jul 24 16:32:57.639045 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.638811 2543 scope.go:117] "RemoveContainer" containerID="deb4ddd557dd4b8bb2bd82e22144a8495d6fe5bbb38d01a04e8219a580678164" Jul 24 16:32:57.718068 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.718011 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:32:57.858098 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858066 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954edbe4-2596-4121-8f65-0e386f75619a-istiod-ca-cert\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858108 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-istio-token\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858162 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-socket\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858194 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-certs\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858264 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-envoy\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858304 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2f7w\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-kube-api-access-l2f7w\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858360 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858330 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954edbe4-2596-4121-8f65-0e386f75619a-istio-podinfo\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858711 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858386 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-credential-socket\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858711 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858453 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-data\") pod \"954edbe4-2596-4121-8f65-0e386f75619a\" (UID: \"954edbe4-2596-4121-8f65-0e386f75619a\") " Jul 24 16:32:57.858711 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858612 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:57.858711 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858639 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954edbe4-2596-4121-8f65-0e386f75619a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 16:32:57.858933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858730 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:57.858933 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858864 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:57.859035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858979 2543 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-credential-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.859035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.858997 2543 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954edbe4-2596-4121-8f65-0e386f75619a-istiod-ca-cert\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.859035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.859011 2543 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-socket\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.859035 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.859024 2543 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-workload-certs\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.859864 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.859841 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-data" (OuterVolumeSpecName: "istio-data") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:57.860838 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.860814 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-kube-api-access-l2f7w" (OuterVolumeSpecName: "kube-api-access-l2f7w") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "kube-api-access-l2f7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:57.860920 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.860853 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/954edbe4-2596-4121-8f65-0e386f75619a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 16:32:57.861066 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.861042 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 16:32:57.861066 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.861051 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-istio-token" (OuterVolumeSpecName: "istio-token") pod "954edbe4-2596-4121-8f65-0e386f75619a" (UID: "954edbe4-2596-4121-8f65-0e386f75619a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:32:57.960149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.960113 2543 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-envoy\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.960149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.960144 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2f7w\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-kube-api-access-l2f7w\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.960149 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.960155 2543 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954edbe4-2596-4121-8f65-0e386f75619a-istio-podinfo\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.960400 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.960164 2543 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954edbe4-2596-4121-8f65-0e386f75619a-istio-data\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:57.960400 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:57.960173 2543 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954edbe4-2596-4121-8f65-0e386f75619a-istio-token\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:32:58.645774 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:58.645747 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" Jul 24 16:32:58.646252 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:58.645785 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2" event={"ID":"954edbe4-2596-4121-8f65-0e386f75619a","Type":"ContainerDied","Data":"668c390ae4d9fee3295c7f59cd7da75b45c895dc28630c794d3cc0b68ea78883"} Jul 24 16:32:58.646252 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:58.645852 2543 scope.go:117] "RemoveContainer" containerID="e10c98831099c100762916fbc5e2bcc00f6499a59fcbc615266aed2c325f5345" Jul 24 16:32:58.667385 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:58.667362 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2"] Jul 24 16:32:58.672832 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:58.672808 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5aebfb-openshift-default-57f6ccf5fb-5ckq2"] Jul 24 16:32:58.904115 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:58.904024 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-j8znp_691e4d01-122f-48e0-a3d8-d055cdd619f2/manager/0.log" Jul 24 16:32:59.025189 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:59.025156 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-frnpz_b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb/manager/0.log" Jul 24 16:32:59.043865 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:59.043839 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954edbe4-2596-4121-8f65-0e386f75619a" path="/var/lib/kubelet/pods/954edbe4-2596-4121-8f65-0e386f75619a/volumes" Jul 24 16:32:59.768226 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:59.768196 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-mdmsk_7681e43a-352b-433f-882d-0daeb4f689ac/copy-manifests/0.log" Jul 24 16:32:59.780268 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:32:59.780244 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-mdmsk_7681e43a-352b-433f-882d-0daeb4f689ac/manager/0.log" Jul 24 16:33:00.246483 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:00.246457 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748480-nwsn5_85ad2343-4950-4d8f-89b6-17abe2cb063f/cleanup/0.log" Jul 24 16:33:00.574195 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:00.574106 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-qswrs_a408116f-e35f-49ce-aac4-4deb1762e2b8/postgres/0.log" Jul 24 16:33:02.158704 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:02.158681 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-9ppx8_38aba10a-872b-4c49-a0d0-ceffeb0ed4eb/manager/0.log" Jul 24 16:33:02.538752 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:02.538723 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-njcn4_80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2/manager/0.log" Jul 24 16:33:03.137137 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:03.137105 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb_7e83ef7d-6413-4a7a-ad0a-d19b163e9f31/istio-proxy/0.log" Jul 24 16:33:03.625924 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:03.625893 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/1.log" Jul 24 16:33:03.859602 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:03.859530 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lj4t_9d58e57f-6052-4e74-9cf5-6114eb69024f/payload-processing/0.log" Jul 24 16:33:04.547351 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:04.547323 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tmdc5_f2647b3c-defc-4c44-a46e-9f591b62ba19/main/0.log" Jul 24 16:33:04.669878 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:04.669855 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68ff928k_6b311d69-5b8e-4714-a15e-d4535d10574d/main/0.log" Jul 24 16:33:13.639022 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:13.638992 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-4shzl_b79ecb11-0e25-4a2a-8292-215db1faa83d/global-pull-secret-syncer/0.log" Jul 24 16:33:13.846979 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:13.846946 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zs6lc_b06a7c14-2007-4bf4-b601-d19eda6ac6f2/konnectivity-agent/0.log" Jul 24 16:33:13.912652 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:13.912578 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-210.ec2.internal_43749135b7c22609bc973c10a709b9e9/haproxy/0.log" Jul 24 16:33:18.776373 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:18.776328 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-9ppx8_38aba10a-872b-4c49-a0d0-ceffeb0ed4eb/manager/0.log" Jul 24 16:33:19.010369 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.010339 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-njcn4_80cd8ae4-ead0-4fe8-9f2f-0abafdc169f2/manager/0.log" Jul 24 16:33:19.342858 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.342822 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-glhzr"] Jul 24 16:33:19.343282 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343268 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76f4caf0-f7d0-4222-8b26-d6d2f90a9415" containerName="payload-processing" Jul 24 16:33:19.343328 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343284 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="76f4caf0-f7d0-4222-8b26-d6d2f90a9415" containerName="payload-processing" Jul 24 16:33:19.343328 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343301 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" Jul 24 16:33:19.343328 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343307 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" Jul 24 16:33:19.343328 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343313 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" Jul 24 16:33:19.343328 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343319 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" Jul 24 16:33:19.343328 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343326 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a135a852-932c-4d47-a160-1c6c2f9d2536" containerName="payload-processing" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343335 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="a135a852-932c-4d47-a160-1c6c2f9d2536" containerName="payload-processing" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343362 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343368 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343373 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c62d562-0eca-49c5-a05f-a75c3226f3ef" containerName="payload-pre-processing" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343378 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c62d562-0eca-49c5-a05f-a75c3226f3ef" containerName="payload-pre-processing" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343389 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343394 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343401 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343407 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343476 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343486 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="393fa908-c4be-4b8f-baec-cdb9d9660a77" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343497 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343507 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" Jul 24 16:33:19.343512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343513 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="bed6449f-0f44-4255-9a43-07c7fb5d9224" containerName="istio-proxy" Jul 24 16:33:19.343970 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343520 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="a135a852-932c-4d47-a160-1c6c2f9d2536" containerName="payload-processing" Jul 24 16:33:19.343970 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343526 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="76f4caf0-f7d0-4222-8b26-d6d2f90a9415" containerName="payload-processing" Jul 24 16:33:19.343970 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.343532 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c62d562-0eca-49c5-a05f-a75c3226f3ef" containerName="payload-pre-processing" Jul 24 16:33:19.346585 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.346555 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:19.348961 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.348939 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-pn9ll\"" Jul 24 16:33:19.353808 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.353787 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-glhzr"] Jul 24 16:33:19.371189 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.371162 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f9dz\" (UniqueName: \"kubernetes.io/projected/6e598d12-efd8-4d4c-86b5-088adee43611-kube-api-access-7f9dz\") pod \"kuadrant-operator-catalog-glhzr\" (UID: \"6e598d12-efd8-4d4c-86b5-088adee43611\") " pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:19.472146 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.472116 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f9dz\" (UniqueName: \"kubernetes.io/projected/6e598d12-efd8-4d4c-86b5-088adee43611-kube-api-access-7f9dz\") pod \"kuadrant-operator-catalog-glhzr\" (UID: \"6e598d12-efd8-4d4c-86b5-088adee43611\") " pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:19.480845 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.480814 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f9dz\" (UniqueName: \"kubernetes.io/projected/6e598d12-efd8-4d4c-86b5-088adee43611-kube-api-access-7f9dz\") pod \"kuadrant-operator-catalog-glhzr\" (UID: \"6e598d12-efd8-4d4c-86b5-088adee43611\") " pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:19.657163 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.657085 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:19.798136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.798113 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-glhzr"] Jul 24 16:33:19.799981 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:33:19.799949 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e598d12_efd8_4d4c_86b5_088adee43611.slice/crio-4f214bebf2ab5307fd8e6611d2cb9e8f17bc825500bf0648ac5e40469171e2c8 WatchSource:0}: Error finding container 4f214bebf2ab5307fd8e6611d2cb9e8f17bc825500bf0648ac5e40469171e2c8: Status 404 returned error can't find the container with id 4f214bebf2ab5307fd8e6611d2cb9e8f17bc825500bf0648ac5e40469171e2c8 Jul 24 16:33:19.801296 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:19.801276 2543 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 16:33:20.763286 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.763257 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" event={"ID":"6e598d12-efd8-4d4c-86b5-088adee43611","Type":"ContainerStarted","Data":"e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071"} Jul 24 16:33:20.763466 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.763315 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" event={"ID":"6e598d12-efd8-4d4c-86b5-088adee43611","Type":"ContainerStarted","Data":"4f214bebf2ab5307fd8e6611d2cb9e8f17bc825500bf0648ac5e40469171e2c8"} Jul 24 16:33:20.780165 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.780123 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" podStartSLOduration=1.4536786369999999 podStartE2EDuration="1.780108255s" totalImagesPullingTime="326.429618ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 16:33:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:33:20.777804153 +0000 UTC m=+3132.309391767" watchObservedRunningTime="2026-07-24 16:33:20.780108255 +0000 UTC m=+3132.311695868" Jul 24 16:33:20.824420 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.824389 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/alertmanager/0.log" Jul 24 16:33:20.851787 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.851716 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/config-reloader/0.log" Jul 24 16:33:20.873642 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.873615 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/kube-rbac-proxy-web/0.log" Jul 24 16:33:20.900220 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.900189 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/kube-rbac-proxy/0.log" Jul 24 16:33:20.922456 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.922422 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/kube-rbac-proxy-metric/0.log" Jul 24 16:33:20.949135 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.949116 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/prom-label-proxy/0.log" Jul 24 16:33:20.971348 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:20.971323 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_06d847ab-a11e-47df-92c1-b926627962ab/init-config-reloader/0.log" Jul 24 16:33:21.050472 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.050443 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dg84v_f264b9b3-6bc6-4dba-98d9-e29aca740f2d/kube-state-metrics/0.log" Jul 24 16:33:21.066573 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.066520 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dg84v_f264b9b3-6bc6-4dba-98d9-e29aca740f2d/kube-rbac-proxy-main/0.log" Jul 24 16:33:21.085657 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.085634 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-dg84v_f264b9b3-6bc6-4dba-98d9-e29aca740f2d/kube-rbac-proxy-self/0.log" Jul 24 16:33:21.113421 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.113360 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-d4965f97d-dgcj7_ca07aae5-3ae0-4fc3-b01f-7a29bf147094/metrics-server/0.log" Jul 24 16:33:21.135541 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.135514 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-ktwhv_dc265e8a-e941-44fe-8fa7-02b49b3ac250/monitoring-plugin/0.log" Jul 24 16:33:21.243460 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.243433 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbs4d_9adb9c66-a5b2-44dc-aa3f-6efc22df5309/node-exporter/0.log" Jul 24 16:33:21.269433 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.269412 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbs4d_9adb9c66-a5b2-44dc-aa3f-6efc22df5309/kube-rbac-proxy/0.log" Jul 24 16:33:21.292994 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.292968 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-sbs4d_9adb9c66-a5b2-44dc-aa3f-6efc22df5309/init-textfile/0.log" Jul 24 16:33:21.401343 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.401247 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-8cpf8_6c5cb812-a998-476f-9131-3d275a6981ef/kube-rbac-proxy-main/0.log" Jul 24 16:33:21.420862 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.420825 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-8cpf8_6c5cb812-a998-476f-9131-3d275a6981ef/kube-rbac-proxy-self/0.log" Jul 24 16:33:21.441572 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.441538 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-b8f49f5c6-8cpf8_6c5cb812-a998-476f-9131-3d275a6981ef/openshift-state-metrics/0.log" Jul 24 16:33:21.481805 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.481781 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/prometheus/0.log" Jul 24 16:33:21.510310 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.510284 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/config-reloader/0.log" Jul 24 16:33:21.532747 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.532726 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/thanos-sidecar/0.log" Jul 24 16:33:21.554201 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.554183 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/kube-rbac-proxy-web/0.log" Jul 24 16:33:21.581153 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.581135 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/kube-rbac-proxy/0.log" Jul 24 16:33:21.601423 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.601405 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/kube-rbac-proxy-thanos/0.log" Jul 24 16:33:21.624711 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.624694 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_7851499f-f916-47c4-8b6c-fe1e9a120c0b/init-config-reloader/0.log" Jul 24 16:33:21.651470 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.651394 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-v5jqd_02c7433e-a5ba-4ab1-aa74-ffd5435935ef/prometheus-operator/0.log" Jul 24 16:33:21.672694 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.672676 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-v5jqd_02c7433e-a5ba-4ab1-aa74-ffd5435935ef/kube-rbac-proxy/0.log" Jul 24 16:33:21.697554 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.697538 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-87vp5_fbfe7372-ca52-41b6-808c-40a9c0003f8e/prometheus-operator-admission-webhook/0.log" Jul 24 16:33:21.731779 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.731757 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7f9fccb9-n4qzn_120bd707-413d-430c-ab1a-94dab7beeb21/telemeter-client/0.log" Jul 24 16:33:21.753547 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.753527 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7f9fccb9-n4qzn_120bd707-413d-430c-ab1a-94dab7beeb21/reload/0.log" Jul 24 16:33:21.776908 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.776870 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-c7f9fccb9-n4qzn_120bd707-413d-430c-ab1a-94dab7beeb21/kube-rbac-proxy/0.log" Jul 24 16:33:21.808424 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.808388 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c5d7468d-dg6nr_df152280-e06e-42df-b5ed-0ee10b3143ab/thanos-query/0.log" Jul 24 16:33:21.828834 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.828814 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c5d7468d-dg6nr_df152280-e06e-42df-b5ed-0ee10b3143ab/kube-rbac-proxy-web/0.log" Jul 24 16:33:21.849173 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.849132 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c5d7468d-dg6nr_df152280-e06e-42df-b5ed-0ee10b3143ab/kube-rbac-proxy/0.log" Jul 24 16:33:21.868102 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.868084 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c5d7468d-dg6nr_df152280-e06e-42df-b5ed-0ee10b3143ab/prom-label-proxy/0.log" Jul 24 16:33:21.889512 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.889495 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c5d7468d-dg6nr_df152280-e06e-42df-b5ed-0ee10b3143ab/kube-rbac-proxy-rules/0.log" Jul 24 16:33:21.909801 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:21.909719 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5c5d7468d-dg6nr_df152280-e06e-42df-b5ed-0ee10b3143ab/kube-rbac-proxy-metrics/0.log" Jul 24 16:33:23.786386 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.786351 2543 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt"] Jul 24 16:33:23.786885 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.786866 2543 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" Jul 24 16:33:23.786966 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.786888 2543 state_mem.go:107] "Deleted CPUSet assignment" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" Jul 24 16:33:23.787148 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.787117 2543 memory_manager.go:368] "RemoveStaleState removing state" podUID="954edbe4-2596-4121-8f65-0e386f75619a" containerName="istio-proxy" Jul 24 16:33:23.791342 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.791326 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.794395 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.794375 2543 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-7s9sw\"/\"default-dockercfg-fh7j9\"" Jul 24 16:33:23.794678 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.794656 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s9sw\"/\"openshift-service-ca.crt\"" Jul 24 16:33:23.795155 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.795143 2543 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-7s9sw\"/\"kube-root-ca.crt\"" Jul 24 16:33:23.800467 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.800446 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt"] Jul 24 16:33:23.813698 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.813678 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-sys\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.813786 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.813725 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-lib-modules\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.813823 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.813791 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-proc\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.813877 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.813828 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-podres\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.813935 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.813919 2543 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59kh\" (UniqueName: \"kubernetes.io/projected/7a973ec8-03bd-4c9c-a7e4-92c12de57005-kube-api-access-s59kh\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914346 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914316 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-proc\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914346 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914349 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-podres\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914390 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59kh\" (UniqueName: \"kubernetes.io/projected/7a973ec8-03bd-4c9c-a7e4-92c12de57005-kube-api-access-s59kh\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914426 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-sys\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914437 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-proc\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914456 2543 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-lib-modules\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914476 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-podres\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914734 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914524 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-sys\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.914734 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.914605 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a973ec8-03bd-4c9c-a7e4-92c12de57005-lib-modules\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:23.922208 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:23.922191 2543 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59kh\" (UniqueName: \"kubernetes.io/projected/7a973ec8-03bd-4c9c-a7e4-92c12de57005-kube-api-access-s59kh\") pod \"perf-node-gather-daemonset-pqhqt\" (UID: \"7a973ec8-03bd-4c9c-a7e4-92c12de57005\") " pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:24.103435 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.103346 2543 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:24.239172 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.239145 2543 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt"] Jul 24 16:33:24.241089 ip-10-0-143-210 kubenswrapper[2543]: W0724 16:33:24.241059 2543 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7a973ec8_03bd_4c9c_a7e4_92c12de57005.slice/crio-ad1b553ac34591a015db86dce15143276c497c2949c53eaba9394ae7ba49b802 WatchSource:0}: Error finding container ad1b553ac34591a015db86dce15143276c497c2949c53eaba9394ae7ba49b802: Status 404 returned error can't find the container with id ad1b553ac34591a015db86dce15143276c497c2949c53eaba9394ae7ba49b802 Jul 24 16:33:24.320030 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.320004 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-8b555fb48-gf8p4_ff644d32-acac-4aa5-ac75-244de09a7d2e/console/0.log" Jul 24 16:33:24.355423 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.355354 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-dpctg_3de9d359-9c0f-4c49-ba30-42b7bbd3fbb7/download-server/0.log" Jul 24 16:33:24.786578 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.786527 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" event={"ID":"7a973ec8-03bd-4c9c-a7e4-92c12de57005","Type":"ContainerStarted","Data":"259e39dc3c0106b845bf0062cb2d940250310519a6a9794a074f873b2bd23646"} Jul 24 16:33:24.786980 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.786601 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" event={"ID":"7a973ec8-03bd-4c9c-a7e4-92c12de57005","Type":"ContainerStarted","Data":"ad1b553ac34591a015db86dce15143276c497c2949c53eaba9394ae7ba49b802"} Jul 24 16:33:24.803845 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:24.803780 2543 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" podStartSLOduration=1.803762463 podStartE2EDuration="1.803762463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 16:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 16:33:24.801502077 +0000 UTC m=+3136.333089709" watchObservedRunningTime="2026-07-24 16:33:24.803762463 +0000 UTC m=+3136.335350078" Jul 24 16:33:25.792137 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:25.792106 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:26.073023 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:26.072950 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n9kqc_3e87160a-8c3e-4880-acd9-409cf1515c2d/dns/0.log" Jul 24 16:33:26.093428 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:26.093399 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n9kqc_3e87160a-8c3e-4880-acd9-409cf1515c2d/kube-rbac-proxy/0.log" Jul 24 16:33:26.162184 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:26.162156 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7clmb_8f7f0e84-4122-475c-bc3d-a2094890f618/dns-node-resolver/0.log" Jul 24 16:33:26.736272 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:26.736244 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-knhkb_a7066765-5a23-4d0f-ac2d-ef3d0173cd65/node-ca/0.log" Jul 24 16:33:27.670178 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:27.670145 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb_7e83ef7d-6413-4a7a-ad0a-d19b163e9f31/istio-proxy/0.log" Jul 24 16:33:27.894605 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:27.894554 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/0.log" Jul 24 16:33:27.923154 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:27.923082 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/1.log" Jul 24 16:33:27.982481 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:27.982455 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lj4t_9d58e57f-6052-4e74-9cf5-6114eb69024f/payload-processing/0.log" Jul 24 16:33:28.774324 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:28.774293 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-84hld_bc12ea4f-c891-41da-8174-b8f91d8b961e/serve-healthcheck-canary/0.log" Jul 24 16:33:29.293126 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.293097 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7tr45_b1e25853-89ba-4910-a9f7-d26e38ae3370/kube-rbac-proxy/0.log" Jul 24 16:33:29.312767 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.312738 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7tr45_b1e25853-89ba-4910-a9f7-d26e38ae3370/exporter/0.log" Jul 24 16:33:29.331944 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.331920 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-7tr45_b1e25853-89ba-4910-a9f7-d26e38ae3370/extractor/0.log" Jul 24 16:33:29.657475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.657389 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:29.657475 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.657454 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:29.680288 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.680264 2543 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:29.835657 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.835633 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:29.904884 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:29.904852 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-glhzr"] Jul 24 16:33:31.502396 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:31.502358 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-j8znp_691e4d01-122f-48e0-a3d8-d055cdd619f2/manager/0.log" Jul 24 16:33:31.701436 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:31.701405 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-frnpz_b6be9a8e-6d4a-4733-97c6-e4fc4e7386eb/manager/0.log" Jul 24 16:33:31.809020 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:31.808950 2543 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-7s9sw/perf-node-gather-daemonset-pqhqt" Jul 24 16:33:31.824709 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:31.824676 2543 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" podUID="6e598d12-efd8-4d4c-86b5-088adee43611" containerName="registry-server" containerID="cri-o://e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071" gracePeriod=2 Jul 24 16:33:32.080133 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.080107 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:32.160976 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.160950 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-mdmsk_7681e43a-352b-433f-882d-0daeb4f689ac/manager/0.log" Jul 24 16:33:32.179025 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.178998 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6798866cc5-mdmsk_7681e43a-352b-433f-882d-0daeb4f689ac/copy-manifests/0.log" Jul 24 16:33:32.195674 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.195651 2543 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7f9dz\" (UniqueName: \"kubernetes.io/projected/6e598d12-efd8-4d4c-86b5-088adee43611-kube-api-access-7f9dz\") pod \"6e598d12-efd8-4d4c-86b5-088adee43611\" (UID: \"6e598d12-efd8-4d4c-86b5-088adee43611\") " Jul 24 16:33:32.197923 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.197892 2543 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e598d12-efd8-4d4c-86b5-088adee43611-kube-api-access-7f9dz" (OuterVolumeSpecName: "kube-api-access-7f9dz") pod "6e598d12-efd8-4d4c-86b5-088adee43611" (UID: "6e598d12-efd8-4d4c-86b5-088adee43611"). InnerVolumeSpecName "kube-api-access-7f9dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 16:33:32.297352 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.297317 2543 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f9dz\" (UniqueName: \"kubernetes.io/projected/6e598d12-efd8-4d4c-86b5-088adee43611-kube-api-access-7f9dz\") on node \"ip-10-0-143-210.ec2.internal\" DevicePath \"\"" Jul 24 16:33:32.830893 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.830864 2543 generic.go:356] "Generic (PLEG): container finished" podID="6e598d12-efd8-4d4c-86b5-088adee43611" containerID="e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071" exitCode=0 Jul 24 16:33:32.831292 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.830904 2543 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" Jul 24 16:33:32.831292 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.830960 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" event={"ID":"6e598d12-efd8-4d4c-86b5-088adee43611","Type":"ContainerDied","Data":"e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071"} Jul 24 16:33:32.831292 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.831010 2543 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-glhzr" event={"ID":"6e598d12-efd8-4d4c-86b5-088adee43611","Type":"ContainerDied","Data":"4f214bebf2ab5307fd8e6611d2cb9e8f17bc825500bf0648ac5e40469171e2c8"} Jul 24 16:33:32.831292 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.831031 2543 scope.go:117] "RemoveContainer" containerID="e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071" Jul 24 16:33:32.841447 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.841428 2543 scope.go:117] "RemoveContainer" containerID="e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071" Jul 24 16:33:32.841718 ip-10-0-143-210 kubenswrapper[2543]: E0724 16:33:32.841699 2543 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071\": container with ID starting with e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071 not found: ID does not exist" containerID="e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071" Jul 24 16:33:32.841785 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.841726 2543 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071"} err="failed to get container status \"e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071\": rpc error: code = NotFound desc = could not find container \"e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071\": container with ID starting with e72abced48eba073c2790b45488d9f611713b661441a7fc649d1145fc9d5f071 not found: ID does not exist" Jul 24 16:33:32.851597 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.851575 2543 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-glhzr"] Jul 24 16:33:32.853641 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:32.853624 2543 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-glhzr"] Jul 24 16:33:33.043616 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:33.043589 2543 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e598d12-efd8-4d4c-86b5-088adee43611" path="/var/lib/kubelet/pods/6e598d12-efd8-4d4c-86b5-088adee43611/volumes" Jul 24 16:33:33.335063 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:33.335039 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-65dbfbfd5-v9wch_32ebc5e8-20f8-449d-8868-c52e760e578f/manager/0.log" Jul 24 16:33:33.984160 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:33.984129 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb_7e83ef7d-6413-4a7a-ad0a-d19b163e9f31/istio-proxy/0.log" Jul 24 16:33:34.191622 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:34.191521 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/0.log" Jul 24 16:33:34.219874 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:34.219846 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/1.log" Jul 24 16:33:34.266874 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:34.266851 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lj4t_9d58e57f-6052-4e74-9cf5-6114eb69024f/payload-processing/0.log" Jul 24 16:33:35.726412 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:35.726377 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-l9csb_7e83ef7d-6413-4a7a-ad0a-d19b163e9f31/istio-proxy/0.log" Jul 24 16:33:35.933637 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:35.933607 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/0.log" Jul 24 16:33:35.961763 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:35.961730 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-nj8hk_54fe0f77-6158-42c0-b519-77cfc6949af3/istio-proxy/1.log" Jul 24 16:33:36.006525 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:36.006455 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-8lj4t_9d58e57f-6052-4e74-9cf5-6114eb69024f/payload-processing/0.log" Jul 24 16:33:39.955136 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:39.955106 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/kube-multus-additional-cni-plugins/0.log" Jul 24 16:33:39.974362 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:39.974335 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/egress-router-binary-copy/0.log" Jul 24 16:33:39.992370 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:39.992350 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/cni-plugins/0.log" Jul 24 16:33:40.010023 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.010003 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/bond-cni-plugin/0.log" Jul 24 16:33:40.028081 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.028056 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/routeoverride-cni/0.log" Jul 24 16:33:40.048716 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.048695 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/whereabouts-cni-bincopy/0.log" Jul 24 16:33:40.066854 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.066834 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-lfd8c_f819a2b1-b2c5-4b6a-b823-af2ef15ba1bd/whereabouts-cni/0.log" Jul 24 16:33:40.350682 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.350643 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mcz5j_777aa0dd-5fc1-4460-b509-b5090e3b07ff/kube-multus/0.log" Jul 24 16:33:40.575162 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.575130 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kmrg6_7be2c164-3700-4df9-a9b6-2e9b0fc2db0e/network-metrics-daemon/0.log" Jul 24 16:33:40.591968 ip-10-0-143-210 kubenswrapper[2543]: I0724 16:33:40.591941 2543 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-kmrg6_7be2c164-3700-4df9-a9b6-2e9b0fc2db0e/kube-rbac-proxy/0.log"